Systems and methods for generating a 3D world model using velocity data of a vehicle

ABSTRACT

A self-contained, low-cost, low-weight guidance system for vehicles is provided. The guidance system can include an optical camera, a case, a processor, a connection between the processor and an on-board control system, and computer algorithms running on the processor. The guidance system can be integrated with a vehicle control system through “plug and play” functionality or a more open Software Development Kit. The computer algorithms re-create 3D structures as the vehicle travels and continuously updates a 3D model of the environment. The guidance system continuously identifies and tracks terrain, static objects, and dynamic objects through real-time camera images. The guidance system can receive inputs from the camera and the onboard control system. The guidance system can be used to assist vehicle navigation and to avoid possible collisions. The guidance system can communicate with the control system and provide navigational direction to the control system.

RELATED APPLICATIONS

This patent application claims priority to U.S. Provisional PatentApplication No. 62/269,557, filed on Dec. 18, 2015, entitled “COLLISIONAVOIDANCE AND GUIDANCE SYSTEM AND DEVICE FOR UNMANNED VEHICLES” and toU.S. Provisional Patent Application No. 62/269,625, filed on Dec. 18,2015, entitled “REAL-TIME VISUAL SITUATIONAL AWARENESS SYSTEM.” Thedisclosures of the above-identified provisional applications areconsidered part of and are incorporated by reference in this patentapplication.

FIELD

The present disclosure relates to a system and device for collisionavoidance and guidance of vehicles.

BACKGROUND

Self-driving cars are taking to the road and autonomous aircrafts arerapidly filling the skies for recreational, commercial and militarypurposes. The aircraft used are autonomous Unmanned Aerial Vehicles(UAVs), meaning their onboard control systems are capable of making realtime decisions independently without relying on human assistance.Usually a nominal flight altitude of 50 m is sufficient to avoidcollision with most static obstacles. However, there could be buildings,trees, wires, telephone poles, and terrain features that cannot bepredetermined through current out-of-date maps and data provided througha route planner. This fact, compounded with the rapid increase in near(100 m) air traffic density, calls for onboard collision avoidancecapabilities that enable UAVs to move around obstacles and otherdynamics objects.

Current systems such as LIDAR and RADAR are used by military and largecommercial or research aircraft. However, these are often too expensivefor smaller UAVs and therefore there is a large and growing need forlow-cost, light-weight solution.

SUMMARY

According to one aspect, a method for generating a 3D world model,includes receiving, by a computing device including one or moreprocessors, from an image capture device mounted on a movable entity, asequence of images including a first image captured at a first time anda plurality of second images captured after the first time, the firstimage of the sequence of images including a first plurality of pixelshaving respective parameter values, the plurality of second images eachincluding a respective second plurality of pixels having respectiveparameter values. The method includes identifying, by the computingdevice, motion data of the movable entity. The method includes tracking,by the computing device, movement of at least a subset of the firstplurality of pixels of the first image across the sequence of images.The method includes estimating, by the computing device responsive totracking the subset of the first plurality of pixels and using thereceived motion data, a depth value for each pixel of the subset of thefirst plurality of pixels. The method also includes generating, by thecomputing device, using the estimated depth value for each pixel of thesubset of the first plurality of pixels, a keyframe to populate a pointcloud including a plurality of points, each point corresponding to arespective pixel of the subset of the first plurality of pixels.

In some implementations, the method includes generating a 3D world modelusing the generated keyframe. In some implementations, identifying, bythe computing device, motion data of the movable entity includesreceiving at least one of velocity data or acceleration data of themovable entity.

In some implementations, the method includes associating, by thecomputing device, to each image of the sequence of images, a respectivevelocity of the movable entity corresponding to a time the image wascaptured by the image capture device.

In some implementations, tracking, by the computing device, movement ofat least a subset of the first plurality of pixels of the first imageacross the sequence of images includes tracking, by the computingdevice, movement of at least a subset of the first plurality of pixelsof the first image across the sequence of images using the respectiveparameter value of the pixel.

In some implementations, estimating, by the computing device, a depthvalue for each pixel of the subset of the first plurality of pixelsincludes estimating the depth value for each pixel of the subset of thefirst plurality of pixels using at least one of velocity data oracceleration data of the movable entity. In some implementations, themethod also includes publishing, by the computing device, a depth mapincluding the estimated depth values for each pixel of the subset of thefirst plurality of pixels of the first image being tracked across thesequence of images, the depth map including, for each tracked pixel,absolute distance information calculated based on a distance the trackedpixel moved across the sequence of images and the velocity data of themovable entity.

In some implementations, the sequence of images includes a firstsequence of images. In some implementations, the method includesreceiving, by the computing device, a second sequence of images afterthe first sequence of images and updating the keyframe responsive toreceiving each image of the second sequence of images. In someimplementations, the keyframe includes a first keyframe and the methodincludes updating, by the computing device, the point cloud with asecond keyframe, the second keyframe generated using the first keyframeand a subset of the second sequence of images captured after the firstkeyframe is generated.

According to one aspect, a system for generating a 3D world modelincludes a hardware processor and a memory coupled to the at least oneprocessor. The system includes an image receiver configured to receive,from an image capture device mounted on a movable entity, a sequence ofimages including a first image captured at a first time and a pluralityof second images captured after the first time, the first image of thesequence of images including a first plurality of pixels havingrespective parameter values, the plurality of second images eachincluding a respective second plurality of pixels having respectiveparameter values. The system includes a motion data receiver configuredto receive motion data of the movable entity. The system includes apixel movement tracker configured to track movement of at least a subsetof the first plurality of pixels of the first image across the sequenceof images. The system includes a depth value estimator configured toestimate, responsive to the tracked subset of the first plurality ofpixels and using the received motion data, a depth value for each pixelof the subset of the first plurality of pixels. The system also includesa keyframe generator configured to generate, using the estimated depthvalue for each pixel of the subset of the first plurality of pixels, akeyframe to populate a point cloud including a plurality of points, eachpoint corresponding to a respective pixel of the subset of the firstplurality of pixels.

In some implementations, the system includes a 3D world model generatorconfigured to generate a 3D world model using the generated keyframe.

In some implementations, the motion data includes at least one ofvelocity data or acceleration data of the movable entity. In someimplementations, the image receiver is configured to associate, to eachimage of the sequence of images, a respective velocity or accelerationof the movable entity corresponding to a time the image was captured bythe image capture device. In some implementations, the pixel movementtracker is configured to track the movement of at least the subset ofthe first plurality of pixels of the first image across the sequence ofimages using the respective parameter values of the subset of the firstplurality of pixels.

In some implementations, the depth value estimator is configured toestimate the depth value for each pixel of the subset of the firstplurality of pixels using velocity data of the movable entity and thekeyframe generator is further configured to publish a depth mapincluding the estimated depth values for each pixel of the subset of thefirst plurality of pixels of the first image being tracked across thesequence of images, the depth map including, for each tracked pixel,absolute distance information calculated based on a distance the trackedpixel moved across the sequence of images and the velocity data of themovable entity, the absolute distance information between the movableentity and the dynamic object.

In some implementations, the sequence of images includes a firstsequence of images and the image receiver is further configured toreceive a second sequence of images after the first sequence of imagesand the keyframe generator is further configured to update the keyframeresponsive to receiving each image of the second sequence of images.

In some implementations, the keyframe includes a first keyframe andwherein the keyframe generator is further configured to update the pointcloud with a second keyframe, the second keyframe generated using thefirst keyframe and a subset of the second sequence of images capturedafter the first keyframe is generated.

According to one aspect, a computer readable storage medium includingcomputer-executable instructions stored thereon, which when executed bya computer, causes the computer to receive, from an image capture devicemounted on a movable entity, a sequence of images including a firstimage captured at a first time and a plurality of second images capturedafter the first time, the first image of the sequence of imagesincluding a first plurality of pixels having respective parametervalues, the plurality of second images each including a respectivesecond plurality of pixels having respective parameter values. Thecomputer is further caused to receive motion data of the movable entity.The computer is further caused to track movement of at least a subset ofthe first plurality of pixels of the first image across the sequence ofimages. The computer is further caused to estimate, responsive totracking the subset of the first plurality of pixels and using thereceived motion data, a depth value for each pixel of the subset of thefirst plurality of pixels. The computer is further caused to generate,using the estimated depth value for each pixel of the subset of thefirst plurality of pixels, a keyframe to populate a point cloudincluding a plurality of points, each point corresponding to arespective pixel of the subset of the first plurality of pixels.

In some implementations, the motion data of the movable entity includesvelocity data of the movable entity. The computer readable storagemedium having further computer-executable instructions stored thereon,which when executed by the computer, causes the computer to associate,to each image of the sequence of images, a respective velocity of themovable entity corresponding to a time the image was captured by theimage capture device.

In some implementations, estimating a depth value for each pixel of thesubset of the first plurality of pixels includes estimating the depthvalue for each pixel of the subset of the first plurality of pixelsusing velocity data of the movable entity. The computer readable storagemedium having further computer-executable instructions stored thereon,which when executed by the computer, causes the computer to publish adepth map including the estimated depth values for each pixel of thesubset of the first plurality of pixels of the first image being trackedacross the sequence of images, the depth map including, for each trackedpixel, absolute distance information calculated based on a distance thetracked pixel moved across the sequence of images and the velocity dataof the movable entity.

In some implementations, the sequence of images includes a firstsequence of images. The computer readable storage medium having furthercomputer-executable instructions stored thereon, which when executed bythe computer, causes the computer to receive a second sequence of imagesafter the first sequence of images and update the keyframe responsive toreceiving each image of the second sequence of images.

According to one aspect, a method for dynamic object tracking includesreceiving, by a computing device including one or more processors, asequence of images captured by an image capture device mounted on amovable entity. The method includes identifying, by the computingdevice, a first image of the sequence of images including a firstplurality of pixels, the first image captured by the image capturedevice at a first time. The method includes identifying, by thecomputing device using one or more image analysis techniques, aplurality of points of interest on the first image, each point ofinterest of the plurality of points of interest mapped to a respectivepixel of the first plurality of pixels. The method includes determining,by the computing device, parameter values for the pixels to which theplurality of points of interest are mapped. The method includesidentifying, by the computing device, across the remaining images of thesequence of images, using the determined parameter values for the pixelsto which the plurality of points of interest identified on the firstimage are mapped, pixels included in the remaining images that haveparameter values that are statistically similar to the determinedparameter values, the identified pixels included in the remaining imagesrepresenting the points of interest identified on the first image. Themethod includes tracking, by the computing device, across the remainingimages of the sequence of images, for each identified point of interest,movement of the point of interest by determining locations of theidentified pixels having the parameter values statistically similar tothe pixel to which the point of interest is mapped in the remainingimages. The method includes determining, by the computing device, aconfidence score of the point of interest indicating a likelihood thatthe point of interest represents a dynamic object by comparing, for eachimage of at least a subset of the remaining images, a distance between afirst location within the image at which the pixel representing thepoint of interest was expected to be located and a second locationwithin the image at which the pixel representing the point of interestwas located. The method includes determining, by the computing device,that at least one point of interest of the plurality of points ofinterest represents a dynamic object responsive to determining that theconfidence score of the point of interest satisfies a threshold value.The method also includes identifying, by the computing device, from atleast one image of the sequence of images, a plurality of pixels thatrepresent the dynamic object represented by the point of interest.

In some implementations, the method includes identifying additionalpoints of interest in the remaining images of the sequence of images andtracking across the remaining images, movement of the additional pointsof interest to identify additional dynamic objects in the sequence ofimages.

In some implementations, the method includes receiving, by the computingdevice, velocity or acceleration data of the movable entity andassociating, by the computing device, to each image of the sequence ofimages, a velocity or acceleration at which the movable entity is movingat the time the image of the sequence of images was captured. In someimplementations, determining the confidence score of the point ofinterest by comparing, for each image of at least a subset of theremaining images, the first location within the image at which the pixelrepresenting the point of interest was expected to be located and thesecond location within the image at which the pixel representing thepoint of interest was located includes determining, by the computingdevice, the first location based on the velocity data of the movableentity.

In some implementations, identifying, by the computing device, from atleast one image of the sequence of images, the plurality of pixels thatrepresent the dynamic object represented by the point of interestincludes identifying, by the computing device, a subset of pixels andtheir corresponding parameter values adjacent to the point of interestrepresenting the dynamic object, tracking, by the computing device, foreach of the subset of pixels, movement of the pixels across a secondsequence of images received by the computing device subsequent to thefirst sequence of images, comparing, by the computing device, across thesecond sequence of images, a displacement of the subset of pixelsadjacent to a point of interest to the movement of the pixelrepresenting the point of interest and identifying, responsive todetermining that the displacement of the subset of pixels isstatistically similar to the movement of the pixel representing thepoint of interest, that the subset of pixels define a shape of themovable entity.

In some implementations, the method includes receiving, by the computingdevice, velocity or acceleration data of the movable entity,determining, by the computing device, a trajectory of the movable entityusing the velocity data of the movable entity, determining, by thecomputing device, based on a trajectory of motion of the dynamic objectrepresented by the point of interest across the sequence of images andthe determined trajectory of the movable entity, that the movable entityand the dynamic object are on a collision path and transmitting, by thecomputing device, data to the movable entity, receipt of the dataconfigured to cause the movable entity to alter the trajectory of themovable entity to avoid a collision with the dynamic object.

In some implementations, the method includes determining, by thecomputing device, that at least one point of interest of the pluralityof points of interest represents a dynamic object responsive todetermining that the confidence score of the point of interest satisfiesa threshold value includes determining, based on the comparing, that thedistance between the first location and the second location exceeds apredetermined minimum distance.

In some implementations, the at least one point of interest includes afirst point of interest and the method further includes determining, bythe computing device, that a second point of interest of the pluralityof points of interest represents a static object responsive todetermining that the confidence score of the second point of interestfails to satisfy a threshold value based on the distance between thefirst location and the second location repeatedly being less than thepredetermined maximum distance.

According to one aspect, a system for dynamic object tracking includes ahardware processor, a memory coupled to the at least one processor, thememory storing instructions, which when executed by the processor,causes the processor to receive a sequence of images captured by animage capture device mounted on a movable entity, identify a first imageof the sequence of images including a first plurality of pixels, thefirst image captured by the image capture device at a first time,identify, using one or more image analysis techniques, a plurality ofpoints of interest on the first image, each point of interest of theplurality of points of interest mapped to a respective pixel of thefirst plurality of pixels, determine parameter values for the pixels towhich the plurality of points of interest are mapped, identify, acrossthe remaining images of the sequence of images, using the determinedparameter values for the pixels to which the plurality of points ofinterest identified on the first image are mapped, pixels included inthe remaining images that have parameter values that are statisticallysimilar to the determined parameter values, the identified pixelsincluded in the remaining images representing the points of interestidentified on the first image, track, across the remaining images of thesequence of images, for each identified point of interest, movement ofthe point of interest by determining locations of the identified pixelshaving the parameter values statistically similar to the pixel to whichthe point of interest is mapped in the remaining images, determine aconfidence score of the point of interest indicating a likelihood thatthe point of interest represents a dynamic object by comparing, for eachimage of at least a subset of the remaining images, a distance between afirst location within the image at which the pixel representing thepoint of interest was expected to be located and a second locationwithin the image at which the pixel representing the point of interestwas located, determine that at least one point of interest of theplurality of points of interest represents a dynamic object responsiveto determining that the confidence score of the point of interestsatisfies a threshold value and identify, from at least one image of thesequence of images, a plurality of pixels that represent the dynamicobject represented by the point of interest.

In some implementations, the processor is further configured to identifyadditional points of interest in the remaining images of the sequence ofimages and track, across the remaining images, movement of theadditional points of interest to identify additional dynamic objects inthe sequence of images.

In some implementations, the processor is further configured to receivevelocity or acceleration data of the movable entity and associate, bythe computing device, to each image of the sequence of images, avelocity or acceleration at which the movable entity is moving at thetime the image of the sequence of images was captured.

In some implementations, to determine the confidence score of the pointof interest by comparing, for each image of at least a subset of theremaining images, the first location within the image at which the pixelrepresenting the point of interest was expected to be located and thesecond location within the image at which the pixel representing thepoint of interest was located, the processor is further configured todetermine the first location based on the velocity data of the movableentity.

In some implementations, to identify, from at least one image of thesequence of images, the plurality of pixels that represent the dynamicobject represented by the point of interest, the processor is furtherconfigured to identify a subset of pixels and their correspondingparameter values adjacent to the point of interest representing thedynamic object, track, for each of the subset of pixels, movement of thepixels across a second sequence of images received by the computingdevice subsequent to the first sequence of images, compare, across thesecond sequence of images, a displacement of the subset of pixelsadjacent to a point of interest to the movement of the pixelrepresenting the point of interest and identify, responsive todetermining that the displacement of the subset of pixels isstatistically similar to the movement of the pixel representing thepoint of interest, that the subset of pixels define a shape of themovable entity.

In some implementations, the processor is further configured to receivevelocity or acceleration data of the movable entity, determine atrajectory of the movable entity using the velocity data of the movableentity, determine, based on a trajectory of motion of the dynamic objectrepresented by the point of interest across the sequence of images andthe determined trajectory of the dynamic object, that the movable entityand the dynamic object are on a collision path; and provide data to themovable entity, receipt of the data configured to cause the movableentity to alter the trajectory of the movable entity to avoid acollision with the dynamic object.

In some implementations, to determine that at least one point ofinterest of the plurality of points of interest represents a dynamicobject responsive to determining that the confidence score of the pointof interest satisfies a threshold value, the processor is furtherconfigured to determine, based on the comparing, that the distancebetween the first location and the second location exceeds apredetermined minimum distance.

In some implementations, the at least one point of interest includes afirst point of interest, the method further comprising determining, bythe computing device, that a second point of interest of the pluralityof points of interest represents a static object responsive todetermining that the confidence score of the second point of interestfails to satisfy a threshold value based on the distance between thefirst location and the second location repeatedly being less than thepredetermined maximum distance.

According to one aspect, a computer readable storage medium includingcomputer-executable instructions stored thereon, which when executed bya computer, causes the computer to receive a sequence of images capturedby an image capture device mounted on a movable entity, identify a firstimage of the sequence of images including a first plurality of pixels,the first image captured by the image capture device at a first time,identify, using one or more image analysis techniques, a plurality ofpoints of interest on the first image, each point of interest of theplurality of points of interest mapped to a respective pixel of thefirst plurality of pixels, determine parameter values for the pixels towhich the plurality of points of interest are mapped, identify, acrossthe remaining images of the sequence of images, using the determinedparameter values for the pixels to which the plurality of points ofinterest identified on the first image are mapped, pixels included inthe remaining images that have parameter values that are statisticallysimilar to the determined parameter values, the identified pixelsincluded in the remaining images representing the points of interestidentified on the first image, track, across the remaining images of thesequence of images, for each identified point of interest, movement ofthe point of interest by determining locations of the identified pixelshaving the parameter values statistically similar to the pixel to whichthe point of interest is mapped in the remaining images, determine aconfidence score of the point of interest indicating a likelihood thatthe point of interest represents a dynamic object by comparing, for eachimage of at least a subset of the remaining images, a distance between afirst location within the image at which the pixel representing thepoint of interest was expected to be located and a second locationwithin the image at which the pixel representing the point of interestwas located, determine that at least one point of interest of theplurality of points of interest represents a dynamic object responsiveto determining that the confidence score of the point of interestsatisfies a threshold value and identify, from at least one image of thesequence of images, a plurality of pixels that represent the dynamicobject represented by the point of interest.

In some implementations, the computer readable storage medium includesfurther instructions, which when executed by the computer, cause thecomputer to identify additional points of interest in the remainingimages of the sequence of images and track, across the remaining images,movement of the additional points of interest to identify additionaldynamic objects in the sequence of images.

In some implementations, the computer readable storage medium includesfurther instructions, which when executed by the computer, cause thecomputer to receive velocity or acceleration data of the movable entityand associate, by the computing device, to each image of the sequence ofimages, a velocity or acceleration at which the movable entity is movingat the time the image of the sequence of images was captured.

In some implementations, to determine the confidence score of the pointof interest by comparing, for each image of at least a subset of theremaining images, the first location within the image at which the pixelrepresenting the point of interest was expected to be located and thesecond location within the image at which the pixel representing thepoint of interest was located, the processor is further configured todetermine the first location based on the velocity data of the movableentity.

According to one aspect, a self-contained, low-cost, low-weight guidancemodule for unmanned vehicles is provided. The guidance module caninclude an optical camera, a case, a processor, a connection between theprocessor and an on-board control system of the vehicle, and one or morecomputer algorithms running on the processor. The guidance module can beintegrated with the control system of the vehicle through “plug andplay” functionality or a more open Software Development Kit. Thecomputer algorithms can be configured to cause the processor of theguidance module to re-create 3D structures as the vehicle travels andcontinuously update a 3D model of the environment. The guidance moduleis configured to continuously identify and track terrain, staticobjects, and dynamic objects through real-time camera images. Theguidance module can receive inputs from a camera mounted on or otherwiseconfigured to move with the vehicle and an onboard control system. Theguidance module can be used to assist vehicle navigation and to avoidpossible collisions. The guidance module is configured to establish oneor more connections between the guidance module and the control systemor an interface to the control system for sending commands directly tothe control system or the interface, and providing navigationaldirection to the control system. In some implementations, the vehiclecan be a drone, a car, a truck, a water vehicle, an aerial vehicle orany other movable entity that has a control system that can control themovement of the movable entity. In some implementations, the controlsystem can be a flight controller or an autopilot module of the vehicle.

The guidance module is configured to recognize environments from animage stream and is able to optimize for various environmentalconditions for more effective interpretation. A mapping solution isprovided whereby monocular camera images are translated and correlatedin 3D-space and over time to create near real-time depth maps of theenvironment. The guidance module can measure confidence at each trackedpixel and uses this to be able to track dynamic and static objects.Additionally, dynamic objects are identified using filtering techniquesand prominent objects are tracked across consecutive frames of the imagestream. From the identified dynamic and static objects, the distance anddepth to each object can be calculated by the guidance module. With thecalculated 3D information of each identifiable object in theenvironment, the guidance module can be used for applications such ascollision avoidance to keep unmanned vehicles safe during operations.

Using a purely electro-optical sensing approach, a very robustunderstanding of the environment can be achieved. This approach allowsthe guidance module to be implemented at a very low monetary cost anddoes not require very much power as it is a passive sensing system.

According to one aspect, a device for using camera images in a guidancesystem includes a processor; a camera; an enclosure; and a dataconnection to a control system of a movable entity. The device receivescamera images from the camera and processes the images to generate a 3Dworld model, track dynamic movement, and update a situational awarenessmodel for communicating collision avoidance instructions to the controlsystem.

According to one aspect, a navigation and safety tool for an unmannedvehicle includes a method for sensing and avoiding terrain, obstacles,and dynamic objects in the path of motion, and a method for navigatingaround obstacles to provide uninterrupted operation.

According to one aspect, a self-contained low-cost, low-weight collisionavoidance module includes a continuous stream of live images, onboardprocessing to create real-time 3D environment maps and to locate othermoving objects, an intuitive application programming interface, and anavigation solution for any unmanned vehicle.

According to one aspect, a modular design allowing ultimate systemflexibility, includes an interchangeable camera module, aninterchangeable processing unit, and continuously or periodicallyupdated algorithms providing more efficient visual interpretation.

According to one aspect, a unique control system interface includes astream of 3D point clouds, a stream of depth maps of the current frameand a continuous navigational command. In some implementations, depthmaps include 2D depth information relative to a specific camera pose. Insome implementations, 3D point clouds can include 3D structuresgenerated from a single seed point at which a system, such as theguidance system, was initialized.

According to one aspect, a real-time visual situational awareness systemincludes a continuous stream of live images, an unsupervised scenefiltering and structuring algorithm, a system to create real-time 3Dstatic maps, a confidence measurement algorithm and dynamic objecttracking for full environment perception.

According to one aspect, a unique scene filtering and structuringalgorithm includes a scene recognition and quality estimator, analgorithm to extract appropriate information about the currentenvironment, and an unsupervised machine learning technique for matchingscene qualities and creating appropriate classifiers.

According to one aspect, a system for creating real-time 3D mapsincludes a simultaneous localization and mapping algorithm based onmonocular optical flow, and confidence measurement algorithm to be ableto extract uncertain areas of the depth map.

According to one aspect, a dynamic object tracker for full environmentperception, comprising a blob filter algorithm, a contrast ratio filterand a kalman filter for blending direct measurements with indirectconfidence measurements. In some implementations, the dynamic objecttracker for full environment perception can include one or more of apoint of interest indicator, a feature descriptor, a corner detector,among others. In some implementations, the dynamic object tracker forfull environment perception can include an impulse response filter orinfinite response filter a sensor fusion filter.

According to one aspect, a dynamic object tracker for full environmentperception can include an initial position seed, a position trackingalgorithm and a trajectory generation algorithm.

BRIEF DESCRIPTION OF THE DRAWINGS

Embodiments of the present disclosure will now be described, by way ofexample only, with reference to the attached Figures, wherein:

FIG. 1 depicts a perspective view of a guidance device, self-containedand enclosed, according to a non-limiting embodiment;

FIG. 2 depicts a perspective view of the device of FIG. 1 with caseopen, showing the components of the processor, according to anon-limiting embodiment;

FIG. 3 is a schematic diagram of the components of a guidance systemincluding the guidance device of FIG. 1, according to a non-limitingembodiment;

FIG. 4 depicts a perspective view of the device of FIG. 1 mounted on anunmanned vehicle, according to a non-limiting embodiment;

FIG. 5 is a schematic diagram of the device in FIG. 1 in an environmentfor 3D mapping terrain and objects, according to a non-limitingembodiment;

FIG. 6 is a schematic diagram of the device in FIG. 1 in an environmentperforming an avoidance maneuver, according to a non-limitingembodiment;

FIG. 7A is a perspective view of the device in FIG. 1 demonstrating themodularity of the system, according to a non-limiting embodiment;

FIG. 7B is a perspective exploded view of the device shown in FIG. 7Ademonstrating the modularity of the system, according to a non-limitingembodiment;

FIG. 8 is a block diagram illustrating components of a real time visualsituational awareness system, according to a non-limiting embodiment;

FIG. 9A is a block diagram illustrating a system architecture of thereal time visual situational awareness system incorporated in the deviceshown in FIG. 1, according to a non-limiting embodiment;

FIG. 9B is a block diagram illustrating a detailed flow of the real timevisual situational awareness system incorporated in the device shown inFIG. 1, according to a non-limiting embodiment;

FIG. 10 is a flowchart depicting an overview for updating a static 3Dworld model, according to a non-limiting embodiment;

FIG. 11 is a block diagram depicting components of a 3D world mapmanager used in the real time visual situational awareness system shownin FIG. 8, according to a non-limiting embodiment;

FIG. 12 is a flowchart depicting a method for updating a static 3D worldmodel, according to a non-limiting embodiment;

FIG. 13 is a flowchart depicting an overview for updating a dynamicobject tracking model, according to a non-limiting embodiment;

FIG. 14 is a block diagram depicting components of a dynamic objectidentifier used in the real time visual situational awareness systemshown in FIG. 8, according to a non-limiting embodiment;

FIG. 15 is a flowchart depicting a method for updating a dynamic objecttracking model, according to a non-limiting embodiment;

FIG. 16 is a flowchart depicting an overview for updating a real-timesituational awareness model, according to a non-limiting embodiment;

FIG. 17 is a block diagram depicting components of a situationalawareness manager used in the real time visual situational awarenesssystem shown in FIG. 8, according to a non-limiting embodiment;

FIG. 18 is a flowchart depicting a method for updating a real-timesituational awareness model, according to a non-limiting embodiment;

FIG. 19 is a block diagram depicting components of an object trajectorymanager used in the real time visual situational awareness system shownin FIG. 8, according to a non-limiting embodiment;

FIG. 20 is a flowchart depicting a method for generating a predictedtrajectory of an object, according to a non-limiting embodiment;

FIG. 21 is a flowchart depicting an overview for determining ifcollision avoidance instructions are needed, according to a non-limitingembodiment;

FIG. 22 is a block diagram depicting components of a control signalmotion mapper used in the real time visual situational awareness systemshown in FIG. 8, according to a non-limiting embodiment;

FIG. 23 is a flowchart for mapping control signals to vehicle motion,according to a non-limiting embodiment;

FIGS. 24A and 24B are block diagrams depicting embodiments of computingdevices useful in connection with the systems and methods describedherein;

FIGS. 25A and 25B are images of an environment including a dynamicobject moving closer to a movable entity, according to a non-limitingembodiment;

FIG. 26 is an image of an environment identifying points of interest ina terrain and a target indicator indicating a location towards which themovable entity is travelling, according to a non-limiting embodiment;

FIGS. 27A and 27B are images of an environment including a dynamicobject and including a collision avoidance zone indicator indicating azone within which dynamic objects will be classified as potentialthreats, according to a non-limiting embodiment;

FIGS. 28A and 28B are images of an environment including a dynamiccollision avoidance zone indicator indicating a zone within whichdynamic objects will be classified as potential threats, according to anon-limiting embodiment;

FIGS. 29A and 29B are images of an environment including a static objectidentifier that changes color as the trajectory of the movable entityapproaches the static object, according to a non-limiting embodiment;

FIG. 30 is an image of an environment identifying a dynamic object and arange indicator indicating a distance of the dynamic object from themovable entity, according to a non-limiting embodiment.

DETAILED DESCRIPTION

For purposes of reading the description of the various embodimentsbelow, the following descriptions of the sections of the specificationand their respective contents may be helpful:

Section A describes aspects of a guidance device and a guidance system,in accordance with some embodiments;

Section B describes a guidance device for use with control systems of avariety of different vehicles, in accordance with some embodiments;

Section C describes an overview of a situational awareness model forcommunicating collision avoidance instructions to a control system, inaccordance with an embodiment;

Section D describes systems and methods for generating and updating astatic 3D world model, according to a non-limiting embodiment, inaccordance with an embodiment.

Section E describes systems and methods for dynamic object trackingusing a dynamic object tracking model, in accordance with an embodiment;

Section F describes systems and methods for updating a real-timesituational awareness model, in accordance with an embodiment; and

Section G describes systems and methods for trajectory mapping, inaccordance with an embodiment;

Section H describes systems and methods for collision avoidance, inaccordance with an embodiment; and

Section I describes systems and methods for mapping control signals tovehicle motion based on image data, in accordance with an embodiment;and

Section J describes a computing environment which may be useful forpracticing embodiments described herein.

A. Guidance Device and Guidance System

FIG. 1 is a perspective view of a guidance device 100 according to anon-limiting embodiment. FIG. 2 depicts a perspective view of the device100 of FIG. 1 with case open, showing the components of the processor,according to a non-limiting embodiment. The guidance device 100comprises a processor 102, a camera 104, a connection 106 (not shown), acase or housing 108, and a mount 110 (not shown). These components allowfor a self-contained, low-cost, low-weight guidance device, and alsocontribute to the modularity of the device, as discussed further belowin FIGS. 7A and 7B. Although the guidance device 100 shows only onecamera 104, in some implementations, the guidance device can includemultiple cameras that can capture images that can be stitched or mergedtogether to form a single image. In some implementations, the guidancedevice can include four cameras. The four cameras may be configured orarranged to capture images that can be stitched or merged together toform a 360 degree field of view. As used herein, an image is not limitedto an image from a single camera, but rather, can include imagescaptured from multiple cameras but stitched or merged together to form asingle image.

The processor 102 can comprise a variety of hardware including, asnon-limiting examples, an ODROID processor, a Raspberry Pi processor, aTEGRA X1 processor, a full computer, dedicated integrated circuit, FPGAor a microprocessor, and can include embedded storage, centralprocessing unit, Random Access Memory, and other components. As anon-limiting example, the processor 102 can comprise the hardwareconfiguration of a duo core 2 gigahertz processor and 2 GB of RandomAccess Memory, but it would be apparent to the person skilled in the artthat other hardware configuration can suffice. In some embodiments, theprocessor 102 can include or be a part of a computing system, such asthe computing system 2100 described below with respect to FIGS. 24A and24B.

The processor 102 is configured to work in conjunction with camera 104for collecting images. As a non-limiting example, the camera 104 canhave a resolution of 5 megapixels, but it would be apparent to theperson skilled in the art that other camera resolutions can suffice.

The connection 106 can comprise a serial connection. As a non-limitingexample, the connection 106 can comprise an RX-TX serial connection or aBluetooth serial connection, but it would be apparent to the personskilled in the art that another form of communication can suffice. Insome embodiments, the connection can include any connection suitable toallow data to be communicated between the processor 102 and the camera104 and can include one or more connections, such as those described inrelation to the computing system 2300 described below with respect toFIGS. 24A and 24B. In some implementations, a universal asynchronousreceiver/transmitter (UART) or a Controller Area Network (CAN) buscontroller can connect to the control system of the vehicle and can sendand receive data and instructions to and from the guidance device 100.

In some implementations, the guidance device can include one or moresensors that can be used to provide input data to the guidance device100. In some implementations, the guidance device 100 can include anaccelerometer for providing accelerometer data to the guidance device.In some implementations, the accelerometer data can be used to verifythe acceleration data received from the control system of the vehicle.In some implementations, the accelerometer data can be used as the inputdata instead of the acceleration data received from the vehicle. In someimplementations, the guidance device can include a gyroscope todetermine position and/or orientation data of the vehicle. In someimplementations, the guidance device can also include other types ofsensors, for example, a sensor that can determine wind speed, which canbe used as an input for determining route guidance and for estimatingtrajectories of the vehicle itself as well as dynamic objects identifiedin the view of the camera

The limited number and size of components necessary contributes to thelow cost and low weight feature of the device 100, which allows thedevice to be mass produced economically.

FIG. 3 is a schematic diagram of the components of a guidance system300, according to a non-limiting embodiment. The guidance system 300comprises the guidance device 100 of FIG. 1, and each of its components,and a control system 150 onboard an unmanned vehicle, for example, thevehicle shown in FIG. 4. In some implementations, the guidance systemmay not include a control system of a vehicle, but rather, may includean interface to communicate with a control system of a vehicle.

The processor 102 is loaded with software 120, which when executed on orby the processor 102, can cause the processor to perform the computeralgorithms for updating the situational awareness model. The software120 can include Simultaneous Localization and Mapping (SLAM) algorithmsto create 3D depth maps of the environment in real-time. These depthmaps are used to be able to determine possible collisions with theterrain or static obstacles in the path of the vehicle. The software canalso include a dynamic tracking algorithm to track dynamic objects andfuse them into the environmental representation, as will be discussedwith reference to FIGS. 13-16 below. This software is able to operate indifferent environments through a classification, filtering and matchingstep to be able to eliminate glare, lighting changes and otherunstructured elements, allowing indoor and outdoor operation. Thisclassification step can include an unsupervised machine learningalgorithm that is trained on a set of unstructured environments.

The guidance device 100 being separate from the control system 150allows the guidance system 300 to act as an emergency stop, safetysystem and navigational redundancy as the unmanned vehicle travelsunaided through the world. The guidance device 100 outputs a stream ofsuggested navigation commands, coded to be interpreted by the controlsystem 150. For example: “stop and hover”, “navigate left 30 degrees”,“slow down and make a 90 degree right turn and proceed”. In someimplementations, the guidance device can output other types of commandsthat can alter the trajectory of the vehicle without influencing thecontrol system itself. For instance, the command can include deploying aparachute or activating some other device. In some implementations, theguidance device may send other types of commands, such as a message tothe ground station/operator, an audible warning or a visual indicatorthat can indicate a potential collision. This navigation aroundobstacles can, in some implementations, be determined through visualserving and extrapolation of the current vehicle position within the 3Denvironment map. The guidance device 100 being separate from the controlsystem 150 allows for the independent verification of vehicle motionpaths for increased safety and navigational accuracy. It should beappreciated that the guidance device 100 can be integral to the vehicleand the control system of the vehicle but still serve as a way toprovide independent verification of the vehicle motion paths.

In one embodiment, the control system 150 can be an autopilot systemassociated with the unmanned vehicle. In another embodiment, the controlsystem 150 can be an external middle layer between the guidance device100 and the autopilot system of the unmanned vehicle. The middle layermay include an intermediate board or layer (for example, a processor orcomputer), which interprets the 3D depth maps in real-time and thenfurther communicates the instructions to the autopilot. To facilitatethe interpretation of the external middle layer, the module has an openSoftware Development Kit (SDK) and Application Programming Interface(API) which allows any programmer to interface with the operations ofthe perception module and have greater control over the navigation andinformation coming from the system. The API may be a part of the SDK butin some implementations, the SDK and the API may be independent.

The inputs from the connection 106 to the guidance device 100 caninclude velocity information from the control system 150 and cameraimages. In some embodiments, the inputs can include position data,acceleration data, or any other type of data that may be used todetermine, derive or otherwise identify velocity information of thevehicle to which the guidance device is connected. The connection 106can also be used to send navigation commands to the control system 150.In one embodiment, a cognitive flight executive onboard the unmannedvehicle makes the decision about how to respond to the information fromthe guidance device 100. In another embodiment, the guidance device 100makes the decision. The processor 102 is sufficient to make navigationdecisions in real-time given the navigation speed of the unmannedvehicle. In some embodiments, the guidance device 100 is on and activethe entire time the unmanned vehicle is in motion.

The device 100 can be configured to be “plug and play” compatible with acontrol system 150 for ease of interchangeability between aerial, space,ground-based unmanned vehicles, water-based unmanned vehicles,amphibious vehicles, among others.

FIG. 4 depicts a perspective view of the device 100 mounted on anunmanned vehicle 400, according to a non-limiting embodiment. In theexample embodiment shown, the unmanned vehicle is an unmanned aerialvehicle with a mount 110 (not shown). It should be appreciated that theguidance device 100 can be used in conjunction with any movable entity,for example, any vehicle or object whose motion can be controlled. Insome embodiments, the vehicle can be a land based vehicle, such as acar, a truck, a van, a motorcycle, among others. In some embodiments,the vehicle can be an aerial vehicle, such as an aircraft, a helicopter,an aerial drone, among others. In some embodiments, the vehicle can be amanned vehicle. In some embodiments, the vehicle can be an unmannedvehicle.

FIG. 5 is a schematic diagram of the device 100 in an environment for 3Dmapping terrain and objects, according to a non-limiting embodiment. Thedevice 100 is capable of recognizing that the static object 502 isstatic and that the dynamic object 504 is moving, as explained ingreater detailed with reference to at least FIGS. 13-16 below.

FIG. 6 is a schematic diagram of the device 100 mounted on an unmannedvehicle 400 in an environment performing an avoidance maneuver,according to a non-limiting embodiment. The device 100 is shownmaneuvering around an object 502 after determining the relativetrajectories of the device 100 and the unmanned vehicle 400, asexplained in greater detailed below.

B. Modular Guidance System for Use with Control Systems of a Variety ofDifferent Vehicles

FIG. 7A is a perspective view of the device in FIG. 1 demonstrating themodularity of the system, according to a non-limiting embodiment. FIG.7B is a perspective exploded view of the device shown in FIG. 7A. Theprocessor 102, camera 104, connection 106 (not shown) and case 108 canbe replaced, and any necessary software modifications can be made inorder to interface with the control system 150 (not shown). The case 108can be arranged to better integrate into a particular unmanned vehicle.

Additional components can be added to the device 100, includingadditional graphics processors, cooling systems, Integrated Circuits,FPGAs, GPS or an upgraded processor or camera (additional components notshown).

As will be described below, the guidance device can be deployed on awide variety of vehicles. In some implementations, the guidance devicecan be configured to interface with one or more systems of the vehicleon which the guidance device is to be deployed. The systems can includea control system of the vehicle through which the guidance device canreceive data as well as transmit instructions to initiate particularactions on the vehicle. In some implementations, the guidance device canbe configured to receive control signals from the control system.

In some implementations, the control system of the vehicle cancommunicate with the guidance device via one or more interfaces. In someimplementations, a universal asynchronous receiver/transmitter (UART) ora Controller Area Network (CAN) bus controller can connect to thecontrol system of the vehicle and can send and receive data andinstructions to and from the guidance device 100. The data can includeaccelerometer readings, gyroscope readings, heading information, speed,velocity or acceleration information, GPS position information, amongothers. In some implementations, the data can include a waypoint listindicating a trajectory of the vehicle, mode information and high levelstate information of the vehicle.

In some implementations, the guidance device, via the visual situationalawareness system (shown in FIG. 8), can make specific API calls tocommunicate with the control system. In some implementations, thecontrol system can push the information to the visual situationalawareness system automatically or responsive to establishing acommunication protocol between the visual situational awareness systemand the control system of the vehicle. In some implementations, thevisual situational awareness system can establish hooks into the controlsystem to give and receive specific commands, instructions, requests, orresponses. In some implementations, the control system can establishhooks into the visual situational awareness system to give and receivespecific commands, instructions, requests, or responses.

Although various embodiments described herein refer to the guidancedevice as being a plug-and-play device that is compatible with a varietyof different vehicles, the present disclosure is not limited to suchembodiments. The present disclosure also contemplates a vehicle systemthat integrates one or more components or portions of the guidancedevice in one or more other subsystems of the vehicle system. In someimplementations, a vehicle can utilize the guidance device to facilitateand effectuate changes in the trajectory of the vehicle, either bysending commands to a control system, or by providing the guidancedevice direct access to control a drive system of the vehicle. Othercombinations or variations of the integration of the guidance devicewith other components of the vehicle are possible and contemplated.

C. Overview of a Situational Awareness Model for Communicating CollisionAvoidance Instructions to a Control System

FIG. 8 is a block diagram showing a real-time visual situationalawareness system 800 configured to communicate with one or more imagecapture devices 855 and a control system 860 of a movable entity, suchas a vehicle. The real-time visual situational awareness system 800 caninclude one or more of an image manager 805, a control system inputmanager 810, a 3D world map manager 815, a dynamic object tracker 820, asituational awareness manager 825, a trajectory manager 830, a collisionavoidance manager 835 and a control signal motion mapper 840. Thereal-time visual situational awareness system 800 can be executed on aprocessor onboard a vehicle. In some implementations, the real-timevisual situational awareness system 800 can be the software 120 loadedon the processor 102 described with respect to FIGS. 1-5. The imagemanager 805, the control system input manager 810, the 3D world mapmanager 815, the dynamic object tracker 820, the situational awarenessmanager 825, the trajectory manager 830, the collision avoidance manager835 and the control signal motion mapper 840 can include or execute atleast one computer program or at least one script. The image manager805, the control system input manager 810, the 3D world map manager 815,the dynamic object tracker 820, the situational awareness manager 825,the trajectory manager 830, the collision avoidance manager 835 and thecontrol signal motion mapper 840 can each be separate components, asingle component, or part of another system. In some implementations,the image manager 805, the control system input manager 810, the 3Dworld map manager 815, the dynamic object tracker 820, the situationalawareness manager 825, the trajectory manager 830, the collisionavoidance manager 835 and the control signal motion mapper 840 caninclude combinations of software and hardware, such as one or moreprocessors configured to execute one or more scripts.

The real-time visual situational awareness system 800 can also includeone or more content repositories or databases 845. The databases 845 canbe local to the real-time visual situational awareness system 800. Insome implementations, the databases 845 can be remote to the real-timevisual situational awareness system 800 but can communicate with thevisual situational awareness system 800 via a network. The databases 845can include images, keyframes, pixel data, pre-stored settings,historical motion related information of the movable entity, amongothers, to facilitate the functionality of the real-time visualsituational awareness system 800.

In some implementations, the one or more image capture devices 855 caninclude a camera, such as the camera 104 shown with respect to FIGS.1-5. The image capture device 855 can be mounted on a movable entity.The real-time visual situational awareness system 800 can include orotherwise communicate with multiple image capture devices that cancapture images that can be stitched or merged together to form a singleimage. In some implementations, the real-time visual situationalawareness system 800 can include or otherwise communicate with two,three, four, five, six or more than six cameras. The multiple camerasmay be configured or arranged to capture images that can be stitched ormerged together to form a 360 degree field of view. As used herein, animage is not limited to an image from a single camera, but rather, caninclude images captured from multiple cameras but stitched or mergedtogether to form a single image. The control system 860 can include anauto-pilot module configured to autonomously control the movable entityon which the image capture device 855 is mounted. The control system 860can receive instructions from the real-time visual situational awarenesssystem 800, which when executed by the control system 860, can cause themovable entity to maneuver within an environment or otherwise adjust atrajectory, speed or direction of the movable entity.

The image manager 805 can be configured to receive images 860 a-n(hereinafter image or images 860) captured by the image capture device855 (or multiple image capture devices). The images 860 can be asequence of images 860. The images can form a video stream. The imagesmay be received at a predetermined number of images per unit time. Thepredetermined number of images per unit time may be configurable tomatch the needs of the real-time visual situational awareness system800. In some implementations, the image capture device 855 may transmita continuous video stream and the image manager 805 may identify imagesfrom the video stream. In some implementations, the image manager 805may select every nth frame of a video stream. In some implementations,the image manager 805 may discard the remaining image frames.

The image manager 805 can be configured to identify, from the images orvia an initialization process, a type of camera from which the imagesare being received. In some implementations, the image manager 805 candetermine various types of information from the images or through theinitialization process that can be used by one or more modules of thereal-time visual situational awareness system 800. For instance, theimage manager 805 can determine a field of view of the camera from theimages or receive the information from the camera or a systemcontrolling the camera. In some implementations, the image manager 805can identify or determine a zoom level of the camera, a configurationsetting (for example, light setting, etc.) of the camera according towhich the images are being captured, a frame rate, a resolution of thecamera, a size of the images, among others.

The image manager 805 can be configured to process the images to match adesired image type that can be used by the other modules of thereal-time visual situational awareness system 800. For instance, theimage manager 805 can adjust a resolution of the images, adjust a sizeof the images, adjust a color intensity of the images, among others.

The image manager 805 can be configured to apply one or more filters tothe images. In some implementations, the image manager 805 can apply oneor more of a contrast ratio filter, a blob filter, or a kalman filter tothe images. In some implementations, the image manager 805 can apply anobject recognition function on the images to identify various objectsfrom the image. In some implementations, the object recognition functioncan apply one or more filters to identify the various objects. In someimplementations, the image manager 805 can apply one or more scenerecognition algorithms to identify clouds in the field of view, the sun,as well as other objects that may not interfere with trajectory planningor route guidance of the vehicle but may be relevant for the variousmodules of the real-time visual situational awareness system 800.

The image manager 805 can be configured to associate each of the imageswith a timestamp identifying or otherwise associated with a time atwhich the image was captured. In some implementations, the image managermay assign an identifier to each image. In some implementations, theimage manager may store, in the databases 845, entries for each of theimages. Each entry may include the corresponding image, the timestamp,the assigned identifier, among other information, for instance, a zoomlevel, a resolution, frames per second, light intensity level, colorsaturation level, among others. In some implementations, one or more ofthe control system input manager 810, the 3D world map manager 815, thedynamic object tracker 820, the situational awareness manager 825, thetrajectory manager 830, and the collision avoidance manager 835 canassociate other information with each of the entries.

The control system input manager 810 can be configured to establish acommunication interface through which the control system input manger810 can communicate with the control system 865 of the movable entity.The control system input manager 810 may receive and transmit data tothe control system 865. In some implementations, the control systeminput manager 810 can receive input data from the control system 865.The input data can include data pertaining to control instructions forcontrolling the movement or maneuvering of the movable entity. In someimplementations, the input data can include motion data 870. In someimplementations, the motion data 870 can include position data of themovable entity, velocity data of the movable entity, acceleration dataof the movable entity, or any other data from which velocity data (oracceleration data or position data) of the movable entity can bederived. In some implementations, the input data can be received via aninterface with the control system 865 of the movable entity. In someimplementations in which the real-time visual situational awarenesssystem 800 is a part of or otherwise integrally coupled to a vehicle,the control system input manager 810 may not be needed to establish theinterface as the interfaces may already be established.

The 3D world map manager 815 can be configured to initialize, generateand update a 3D world map. The 3D world map manager 815 can provide the3D world map for use in generating a situational awareness model inconjunction with a dynamic object tracking model that tracks dynamicobjects. Further details regarding the 3D world map manger 815 areprovided with respect to at least FIGS. 10-12.

The dynamic object identifier 820 can be configured to identify dynamicobjects from a sequence of images based on movements of pixels acrossthe sequence of pixels. The dynamic object identifier 820 can generate adynamic object tracking model that can identify a plurality of dynamicobjects identified in the sequence of images. In some implementations,the dynamic objects can be vehicles, birds, animals, persons, or otherobjects that may move of their own volition, or be caused to move viaexternal influences, including the wind, gravity, among others.Additional details regarding the dynamic object identifier are providedbelow with respect to at least FIGS. 13-15.

The situational awareness manager 825 can be configured to generate asituational awareness model. The situational awareness model can be usedfor collision avoidance detection, autonomous navigation, among others.The situational awareness manager 825 can rely on outputs generated bythe 3D world model manager 815, the dynamic object identifier 820 andthe trajectory manager 830. Additional details regarding the dynamicobject identifier are provided below with respect to at least FIGS.16-18.

The trajectory manager 830 can be configured to generate a predictedtrajectory of one or more dynamic objects identified by the dynamicobject identifier 820 as well as the predicted trajectory of the vehicleon which the visual situational awareness system 800 is deployed.Additional details regarding the trajectory manager 830 are providedbelow with respect to at least FIGS. 19 and 20.

The collision avoidance manager 835 can be configured to determinewhether a collision between the vehicle and one or more of the dynamicobjects will occur and responsive to determining that a collision willoccur, output an avoidance maneuver signal that will cause the vehicleto perform a maneuver to avoid the collision. Additional detailsregarding the collision avoidance manager 835 are provided below withrespect to at least FIG. 21.

The control signal motion mapper 840 can be configured to map controlsignals to vehicle motion. In particular, the control signal motionmapper can receive control signals from a control system of the vehicle,determine changes across the sequence of images, determine a maneuver ofthe vehicle based on the determined changes and then correlate thereceived control signals to the maneuver of the vehicle. In this way,the control signal motion mapper can map the effect of certain controlsignals on the maneuverability of the vehicle. Additional detailsregarding the control signal motion mapper 840 are provided below withrespect to at least FIGS. 22 and 23.

FIG. 9A is a flowchart depicting a method 900 for updating a situationalawareness model for communicating collision avoidance instructions to acontrol system, according to a non-limiting embodiment. It is to beemphasized, however, that method 900 need not be performed in the exactsequence as shown, hence the elements of method 900 are referred toherein as “blocks” rather than “steps”.

At block 902, a set of images are sequentially captured. In oneembodiment, the set of images can be captured by the camera 104 on thedevice 100. At block 903, motion data associated with the movable entityon which the guidance device 100 is deployed can be received.

At block 904, a situational awareness model is initialized. The camera104 can capture an initial set of images that are used to generate a 3Dworld model. The initialization of a situational awareness model issimilar to updating a situational awareness model, as described at block904, with the added difference that the situational awareness model isbeing generated with a limited number of images. The added differencecould include algorithmic shortcuts in which confidence is temporarilydecreased in order to create a full map in a small number of steps.

At block 906, a situational awareness model is updated. Updating asituation awareness model comprises blocks 908 and 910, for updating a3D world model and updating a dynamic tracking model, respectively, andfeeding the situational awareness model to a collision avoidance module.Additional details for updating the 3D world model and for updating thedynamic tracking model are described herein in further detail.

At block 908, the 3D world model is updated. The 3D world model isupdated by measuring the distance that each pixel of an image movesframe by frame, if the pixel moved at all. By comparing the rate atwhich the pixel moves to the known rate at which the vehicle istravelling, a 3D map of the environment can be generated. The 3D map ofthe environment can be generated using depth information of each pixel,which can be calculated using the known motion related information ofthe vehicle. In some embodiments, the motion related information caninclude a speed at which the vehicle is moving, a velocity of thevehicle, a displacement of the vehicle, or acceleration of the vehicleor any other data that can be used to derive the velocity of thevehicle. Traditionally, the optical flow method is used to calculate thespeed of an object with the use of a stationary camera, whereas in thissystem, depth information is calculated using the known vehicle (andcamera) motion related information. The process of updating the 3D worldmodel is described in greater detail herein.

At block 910, the dynamic tracking model is updated. The dynamictracking model is updated in two ways. First, if a pixel is movingquickly frame by frame, there is greater confidence that the pixel isrepresenting a dynamic object. Second, a segmentation filter (forexample, a “blob” filter, or any other suitable segmentation filter) isused that checks the grayscale contrast between sets of pixels. Wherethere is greater contrast, there is a greater likelihood that the set ofpixels are representing a dynamic object. This object is then trackedfrom frame to frame, allowing for speed and position measurements to beupdated. The process of updating the dynamic tracking model is describedin greater detail herein.

At block 912, a collision avoidance decision is made. The collisionavoidance decision can be based on a current trajectory of the vehicle.In some implementations, the collision avoidance decision can be basedon a predicted trajectory of one or more dynamic objects tracked via thedynamic tracking model. In some implementations, the collision avoidancedecision is made using the 3D world environment and the relativetrajectories of the vehicle and the dynamic objects also identified inthe 3D world environment. In some implementations, the collisionavoidance decision is made using the 3D world environment and thetrajectory of the vehicle and a static object identified within aparticular position within the 3D world environment.

At block 914, a collision avoidance instruction is communicated to acontrol system such as the control system 150. For example, in practice,after creating the 3D depth maps and tracking dynamic objects, anApplication Programming Interface (API), for example, the API that ispart of the SDK referenced above, could suggest a course of action basedon the object's relative location to the vehicle. This could be thedistance and XYZ position of the closest ten obstacles with which thevehicle may collide, and the confidence intervals associated with theseobstacles in one obstacle information array. The array could berepresented as a set of point cloud models, depth maps, list of trackedobjects or points, or in any other format that may be used for collisionavoidance. The situational awareness system can be configured to run inparallel with the autopilot onboard any vehicle and offer navigationalassistance and/or emergency control of the vehicle. For example: “stopand hover”, “navigate left 30 degrees”, “slow down and make a 90 degreeright turn and proceed”. In some implementations, the situationalawareness system can output other types of commands that can alter thetrajectory of the vehicle without influencing the control system itself.In some implementations, the trajectory of the vehicle can be alteredeven if it does not alter a direction of the vehicle. The vehicle'strajectory can be changed if the vehicle is slowed down or sped up suchthat the vehicle arrives at a particular location at a time differentthan a time at which the vehicle was previously expected to arrive. Forinstance, the command can include deploying a parachute or activatingsome other device. In some implementations, the situational awarenesssystem may send other types of commands, such as an audible warning or avisual indicator that can indicate a potential collision. Additionaldetails regarding each of the blocks 902-814 are provided herein.

FIG. 9B is a block diagram illustrating aspects of the real time visualsituational awareness system incorporated in the device shown in FIG. 1,according to a non-limiting embodiment. In some implementations, thereal-time visual situational awareness system 800 can executeinstructions described with respect to the flow shown in 9B. A startupdaemon process 920 can execute on a processor to launch the real-timevisual situational awareness system 800. A watchdog program 922 cancommunicate with the real-time visual situational awareness system 800.The watchdog program 922 can serve as an interface between one or moreother systems of the movable entity or associated with the movableentity and the real-time visual situational awareness system 800. Acamera port 924 can be communicatively coupled to an image capturedevice, such as the camera 104 and can provide an image stream 926 tothe real-time visual situational awareness system 800. The image stream926 can include a plurality of images. In some implementations, theimage stream 926 can be or can include images from a real-time livevideo feed.

The real-time visual situational awareness system 800 can identify, fromthe image stream 926, each image and analyze the image for generating a3D world model and for identifying dynamic objects. The real-time visualsituational awareness system 800 may analyze multiple images of theimage stream in parallel or may utilize pixel data from multiple imagesto generate the 3D world model or to identify dynamic objects.

The real-time visual situational awareness system 800 can also includean interface 932, such as an auto-pilot interface or a control systeminterface for establishing communications with a communications port 934of an auto-pilot function or control system of the movable entity. Theinterface 932 can enable receiving an autopilot data package 936 fromthe communication port 934 of the auto-pilot function or control systemof the movable entity. The autopilot data package 936 can include one ormore of velocity information 938 or acceleration information 940 of themovable entity. In some implementations, the autopilot data package 936can include any information from which the velocity information 938 orthe acceleration information 940 can be derived or otherwise determinedor estimated.

Referring now to block 930, a 3D world map manager, such as the 3D worldmap manager 815 can perform unsupervised scene recognition on each ofthe images. This can include applying various filters to the images toidentify objects in the image and to classify them as static or dynamicobjects and group pixels that represent a particular object together. Anoptical flow measurement function can be executed on the output of theunsupervised scene recognition function. The optical flow measurementfunction, which can be used to determine depth values can receive thevelocity information 938 of the movable entity to determine the depthvalues of pixels tracked across images 928 of the image stream 926.

The optical flow measurement function can provide pixel position andmovement information in conjunction with the depth value of therespective pixels to the keyframe update function, which can then updatea keyframe based on the received pixel related data. The keyframe updatefunction can update keyframes that are used to build the 3D world model.As more and more images are received and are processed, the 3D worldmodel is improved by adding new pixel related data to the keyframes usedto build the 3D world model.

In a separate but parallel process, the image stream 926 can also beprocessed to identify dynamic objects as described briefly above. Eachimage from the image stream 926 is processed by applying one or moredifferent types of image filters, for example, a blob filter. The blobfilter can be used to detect anomalies in the scene and segment possiblemoving targets. The blob filter can be used to identify sets of pixels(“blobs”) in each image. The image can further be processed by acontrast ratio measurement function that can compare, for each blob, thecontrast of the contents of the blob to pixels surrounding the blob tocreate a contrast ratio quantity. The real-time visual situationalawareness system 800 can provide the contrast ratio quantity to abackground subtraction function that can use the contrast ratio quantityfor discarding objects which are not real threats and simply backgroundnoise. As each image is processed and the pixels are tracked acrossmultiple images, each segmented blob can be given a certainty score thatis based on the confidence measurement and contrast ratio. Eachsegmented blob is tracked to determine velocity, direction and distancefrom the movable entity. Where there is greater contrast, there may be agreater likelihood that the set of pixels is representing a dynamicobject.

A dynamic tracking function performed by the dynamic object identifier820 and/or the trajectory manager 830 can receive pixel data relating tothe blobs and their respective contrast ratios from the backgroundsubtraction function. The dynamic tracking function can identify thepixels corresponding to dynamic objects based on the contrast levels ofthe pixels across the sequence of images. The dynamic tracking functioncan provide pixel information corresponding to the dynamic objects forinclusion into a world model that includes both static objects as wellas the dynamic objects identified by the dynamic tracking function.

The world model that includes both static objects and dynamic objects isupdated as more images are received and processed. The world model isupdated using the acceleration data and velocity data being provided atvarious stages of the world model generation process performed by the 3Dworld map manager 815.

A trajectory propagation function executed by the trajectory manager 830can be configured to generate predicted trajectories of the movableentity and the dynamic objects identified by the dynamic trackingfunction. The trajectory propagation function can aid in identifying orclassifying the dynamic object into one of a plurality of differenttypes of dynamic objects. In some implementations, a dynamic objectclassifier can classify the type of dynamic object. The trajectorypropagation function can, from changes in the world model over time,generate a predicted trajectory for each dynamic object based on boththe motion of the movable entity as well as the motion of the dynamicobject.

The trajectory propagation function can provide trajectory informationof the plurality of dynamic objects and/or the movable entity to acollision detection function. The collision detection function canreceive velocity data 938, which the collision detection function 962can use to determine if a collision is likely to occur between themovable entity and one or more of the dynamic objects based on thetrajectories of the dynamic objects and the velocity of the movableentity. Responsive to the collision detection function detecting apotential collision event, the collision detection function can passinformation to a navigation suggestion function, which can determine,based on the projected trajectories of the dynamic objects and thevelocity of the movable entity, a maneuver the movable entity canperform to avoid the potential collision event. The navigationsuggestion function can then generate one or more instructions toprovide to the interface 932, to communicate with a control system orother system of the movable entity, the generated instruction to causethe movable entity to perform a maneuver to avoid the potentialcollision event.

It should be appreciated that the real-time visual situational awarenesssystem 800 can continuously receive images from the camera port and thereal-time visual situational awareness system 800 can repeatedly processeach of the images to further update the keyframes and further improvethe world model. As such, over time, the world model improves and thereal-time visual situational awareness system 800 produces more accurateworld maps as more data is received and processed.

D. Systems and Methods for Updating a Static 3D World Model

FIG. 10 is a flowchart depicting a method 1000 for updating a static 3Dworld model, according to a non-limiting embodiment. It is to beemphasized, however, that method 1000 need not be performed in the exactsequence as shown, hence the elements of method 1000 are referred toherein as “blocks” rather than “steps”.

At block 1002, an image is captured. The image can be captured by thecamera 104 on the device 100 and/or by image capture device 855. In someimplementations, a monocular image setup is used in order to keep thesystem simple and reduce errors and limits associated with staticbaselines with fixed, multiple cameras. The knowledge of movement can beused in order to create dynamic baselines which extend the range overwhich depth calculations can be made. For example, the disparityinformation from stereo images can be recreated over time with asequence of successive monocular images. At block 1003, motion dataassociated with the movable entity on which the guidance device 100 isdeployed can be received.

At block 1004, the initial keyframe to populate the 3D world model isgenerated.

At block 1006, the 3D world model is stored as a sparse point cloud. The3D world model can be continuously updated with each new keyframe. Eachindividually created depth map can be based on an original keyframe andthe combination of a series of successive images. In order to calculatedepth information, an optical flow algorithm can be employed to be ableto determine individual pixel movement between each frame across thesequence of images

At block 1008, the keyframe is continually updated as each new image iscollected and fed into the keyframe at block 1012. These keyframes canbe matched and also correlated in order to provide a real-time 3D map ofthe space being navigated. In some implementations, a reference keyframeis used and successive frames are tracked in order to simulate largerbaselines and create a more accurate depth map. Once enough translationhas led to an accurate depth map, the map can be added to the 3D worldmodel and a new reference keyframe can be established. The method 1000can allow for 3D depth maps to be published in real-time. In someimplementations, the real-time nature of the method 1000 is importantfor continuous situational awareness, and can advantageously improve theperformance of a system implementing the method 1000 at short range aswell as at long range.

Existing featured SLAM systems may not be able to detect disturbances atthe range that is possible with this situational awareness systembecause they typically utilize stereo vision and/or computer visionfeatures, requiring more pixels and effectively cutting down spatialresolution. At sufficient processing speed, a system implementing method1000 can outperform existing systems at close range because the lack ofcontrast is usually a detriment while this method 1000 needs onlyindividual pixel information for tracking purposes.

Compared to existing photogrammetric range imaging algorithms, a systemimplementing method 1000 can be more robust and can increase thelikelihood that all important elements of the image are being tracked.In some implementations, the featureless nature of the SimultaneousLocalization and Mapping (SLAM) according to the present disclosureallows for better operation at larger distances and in less structuredenvironments than existing approaches. Unlike other SLAM approaches,SLAM according implementations of the present disclosure takes theunique method of adding velocity data, and allows the depth maps to bepublished with absolute distances rather than needing loop closure todetermine real depth. The confidence layer can allow for a uniqueapproach to tracking dynamic objects within the environment whichcomplements the contrast ratio approach, employed in parallel. The SLAMalgorithms can provide a reliable static model and the dynamic trackingallows for changing information in the environment to be followed.Together, this can provide a very robust situational awareness layer forany robotic system.

In order to create absolute depth information, the system can take, asinput, velocity data of the system. This velocity information and timingof the system can allow for a determination of the absolute rate ofchange of obstacles in the environment. The absolute rate of change, acore element of the depth maps created, corresponds directly to theabsolute distances to the various objects which comprise the environmentin the image.

At block 1010, a confidence value is attributed to every pixel that hasmoved between the current keyframe and the previous (reference)keyframe. In some implementations, each additional point is tracked in a3D point cloud, which is continuously being updated with new keyframes.Each new, slightly adjusted point can be added through a position filterto the point cloud so that each new frame adds to the total confidenceof the system. In some implementations, confidence is a localmeasurement dependent on the number of points which appear continuouslyin the same position and is tracked until that confidence decreases tobelow a set value, at which time the point is discarded, freeing upmemory and additional, unnecessary processing. The confidencemeasurements and the overall accuracy of the system can increase as themovable entity approaches any possible objects and is able tocommunicate the threat with increasing accuracy. If a pixel has movedsignificantly, it may be more likely that the pixel represents an objectthat is dynamic. If a pixel has not moved significantly, it may be morelikely that the object represented by the pixel is static.

FIG. 11 is a block diagram illustrating further details of the 3D worldmap manager 815, which can be configured to generate a 3D world model.The 3D world map manager 815 can include a pixel identifier 1105, apixel movement tracker 1110, a depth value estimator 1115, a keyframemanager 1120, and a 3D world map generator 1125. The pixel identifier1105, pixel movement tracker 1110, depth value estimator 1115, keyframemanager 1120, and 3D world map generator 1125 can include or execute atleast one computer program or at least one script, and can includecombinations of software and hardware, such as one or more processorsconfigured to execute one or more scripts.

The 3D world map manager 815 is configured to receive a sequence ofimages 860 a-n. For example, the 3D world map manager 815 can include animage receiver configured to receive the sequence of images. Thesequence of images can be received from an image capture device mountedon a movable entity, such as the image capture device 855. The sequenceof images can include a first image captured at a first time and aplurality of second images captured after the first time. For example,the sequence of images may be associated with sequential times or timestamps, such that the first image is captured at a first time prior towhen the plurality of second images are captured. The image receiver canbe configured to determine the time associated with each image from thesequence of images, such as by extracting the time from metadatatransported with each image.

The first image 860 a of the sequence of images 860 a-n includes a firstplurality of pixels. The plurality of second images 860 b-n also eachinclude a respective second plurality of pixels. Each image can berepresented as an ordered or organized set or plurality of pixels, suchas an arrangement of m-by-n pixels (e.g., m rows of pixels by n columnsof pixels).

Each pixel (e.g., the first plurality of pixels, the respective secondplurality of pixels) has or is associated with one or more respectiveparameter values. The parameter values can store, encode, or otherwiserepresent image information captured in each image. For example, theparameter values may be associated with one or more of a color,intensity, luminance, or other parameter representing image informationor indicative of the light associated with the pixel captured in theimage. In some implementations, each pixel is associated with agreyscale intensity value and one or more values representing color(e.g., representing color in an RGB color format).

The pixel identifier 1105 can be configured to identify the pixels ofthe sequence of images 860 a-n. For example, the pixel identifier 1105can be configured to extract the pixels from the sequence of images 860a-n. The pixel identifier 1105 can be configured to receive an image andextract the pixels from the image, such as by storing parameter valueinformation for each pixel in an array, a map or other data structurecorresponding to each image of the sequence of images 860 a-n.

In some implementations, the pixel identifier 1105 is configured toselectively identify pixels. For example, the pixel identifier 1105 canbe configured to apply a filter to the sequence of images 860 a-n toselectively identify pixels. The filter can be based on one or moreparameter values. For example, the filter can be based on an intensityof each pixel. In some implementations, the pixel identifier 1105 canselectively identify all of the pixels of each of the images.

The 3D world map manager 815 is configured to receive the motion data870. For example, the 3D world map 815 can include a motion datareceiver or tracker configured to receive the motion data 870. Themotion data 870 can include velocity information and/or accelerationinformation regarding the movable entity or any other information fromwhich velocity or acceleration data of the movable entity can bederived, ascertained or otherwise determined. The motion data 870 canalso include attitude data or other data representing a motion state orenergy state of the movable entity (e.g., attitude, thrust, heading,bearing, orientation). The motion data 870 can also include or representa position of the movable entity (e.g., an absolute position, such asGPS coordinates; a relative position such as a position relative to aleader or follower object or a base station).

The 3D world map manager 815 (e.g., a component of the 3D world mapmanager such as the image receiver or the motion data receiver ortracker) can be configured to associate, to each image of the sequenceof images 860 a-n, a respective motion data value of the movable entitycorresponding to a time the image was captured by the image capturedevice 855. In some implementations, each image can be tagged orassociated with a timestamp and a current motion state or status of themovable entity at the time the image was taken. In some implementations,the current motion state can include an absolute or relative position ofthe movable entity, a velocity of the movable entity, an acceleration ofthe movable entity, among others. The position data can be 3D positiondata such that an aerial movable entity can have different position datavalues based on the different altitudes at which the dynamic object isflying. The 3D world map manager 815 can generate and store currentmotion state information regarding the movable entity, to generate aprofile representing position, velocity, or other motion informationover time. The profile can be used by the 3D world map manager 815 todetermine depth information for each image or otherwise generate a modelof the environment about the movable entity, as will be describedfurther herein.

The pixel movement tracker 1110 is configured to track movement ofpixels in or across the sequence of images 860 a-n. The pixel movementtracker 1110 can be configured to track movement of at least a subset ofthe first plurality of pixels of the first image 860 a across thesequence of images 860 a-n. For example, as the movable entity changesin position and/or orientation, pixels may enter or exit the images 860captured by the fixed camera of the movable entity (e.g., image capturedevice 855), such that only a subset of the first plurality of pixels ofthe first image 860 a may be tracked across the sequence of images 860b-n. In some implementations, the pixel movement tracker 1110 can beconfigured to track each of the pixels of an image but only a subset ofthe pixels that were included in a first image remain as the pixelmovement tracker 1110 tracks the same pixel across subsequent images aspixels corresponding to objects (or objects corresponding to the pixels)are no longer in the view of the camera and therefore, not captured.Some pixels in the sequence of images 860 a-n may be associated withfeatures in an environment about the movable entity that may be static(e.g., the absolute positions of the features do not change over time,though the pixels associated with those features in the images 860 a-nmay change as the relative position of the movable entity changes ascompared to the features). Some other pixels in the sequence of images860 a-n may be associated with features in an environment about themovable entity that may be dynamic (e.g., the absolute positions of thefeatures change over time regardless of whether the absolute position ofthe movable entity changes). For example, given an m-by-n arrangement ofpixels, a feature (e.g., a pixel-sized feature as captured by the imagecapture device 855) of the environment may be associated with a pixel inposition [m₁, n₁] of the first image 860 a, and with a pixel in position[m₂,n₂] of the second image 860 b. The pixel movement tracker 1110 canbe configured to track movement of the pixel (e.g., movement of afeature represented by the pixel) from position [m₁, n₁] to position[m₂, n₂] from the first image 860 a to the second image 860 b. The pixelmovement tracker 1110 can further compute a distance the pixel movedbased on a difference between the coordinates of [m₁, n₁] and [m₂,n₂].

Referring briefly to FIGS. 25A and 25B are images of an environmentincluding a dynamic object moving closer to a movable entity, accordingto a non-limiting embodiment. In some implementations, the pixel canrepresent a dynamic object such as a plane. As shown in FIGS. 25A and25B, a plane 2502 represented by the pixel moves from a first pixelposition of the first image 2500A to a second pixel position of thesecond image 2500B. For frame of reference, a static object shown as2504 remains in approximately the same pixel position in the two images2500A and 2500B.

In some implementations, the pixel movement tracker 1110 is configuredto track movement of pixels based on or using parameter values. Forexample, the pixel movement tracker 1110 can track the movement of atleast the subset of the first plurality of pixels of the first image 860a across the sequence of images 860 a-n using the respective parametervalues of the subset of the first plurality of pixels.

The pixel movement tracker 1110 can be configured to track movement ofpixels based on determining a match between a first pixel of the firstplurality of pixels and a second pixel of the second plurality ofpixels. For example, the pixel movement tracker 1110 can identify thefirst pixel based on a first parameter value of the first pixel (e.g., afirst intensity). The pixel movement tracker 1110 can search the secondplurality of pixels to identify or determine one or more second pixelshaving a second parameter value (e.g., a second intensity) correspondingto or matching the first parameter value. In some implementations, thepixel movement tracker 1110 is configured to identify or determine amatch or correspondence based on comparing the first parameter value tothe second parameter value. For example, the pixel movement tracker 1110can determine a difference between the first parameter value and thesecond parameter value, and compare the difference to a threshold value;if the difference is less than a threshold value, then the pixelmovement tracker 1110 can determine that a match or correspondenceexists between the first pixel and second pixel. In someimplementations, the pixel movement tracker 1110 is configured todetermine that a plurality of second pixels have an initial orpreliminary match or correspondence to the first pixel, and furtherrefine the matching to identify a final match or correspondence based onat least one of a greatest similarity (e.g., least difference) betweenthe first pixel and second pixel or an expected location of the secondpixel. In some implementations, the pixel movement tracker 1110 may useparameter values of pixels adjacent to the tracked pixel to facilitateidentifying the tracked pixel across the multiple images. It should beappreciated that one skilled in the art may deploy one or more existingpixel tracking techniques to track the pixels across the sequence ofimages.

The depth value estimator 1115 is configured to estimate a depth valuefor pixels of the first plurality of pixels. In some implementations,the depth value estimator 1115 is configured to estimate, responsive tothe tracked subset of the first plurality of pixels and using thereceived motion data, a depth value for each pixel of the subset of thefirst plurality of pixels. The depth value estimator 1115 can estimatethe depth value based on determining an indication of movement from themotion data. For example, the depth value estimator 1115 can determine amotion state profile over time (e.g., position over time, velocity overtime, attitude over time) of the movable entity, such as by comparingabsolute or relative velocities of the movable entity associated withthe motion data. In some implementations, as the magnitude of a changein the motion state of the movable entity increases between two pointsin time, the depth value estimator 1115 can associate a relativelygreater difference in depth between the images 860 corresponding to thetwo points in time.

The keyframe manager (or generator) 1120 is configured to generate akeyframe to populate a point cloud. For example, the keyframe manager1120 can generate, using the estimated depth value for each pixel of thesubset of the first plurality of pixels, a keyframe to populate thepoint cloud. The point cloud can include a plurality of points. Eachpoint can correspond to a respective pixel of the subset of the firstplurality of pixels. Where the image data (e.g., the subset of the firstplurality of pixels) may be a two-dimensional representation of theenvironment about the movable entity, the keyframe provides a 3Drepresentation by including the depth value for each pixel. For example,the keyframe manager 1120 can apply a transformation to the subset ofthe first plurality of pixels to convert the subset of the firstplurality of pixels from a two-dimension representation (e.g., an m-by-narrangement of pixels which may have corresponding parameter values suchas intensity and color) to a three-dimension representation (e.g., anm-by-n-by-k arrangement of pixels which may have the same correspondingparameter values such as intensity and color but with an additionalassociated depth/distance to each pixel).

In some implementations, the sequence of images 860 a-n includes a firstsequence of images 860 a-m. The 3D world map manager 815 can beconfigured to receive a second sequence of images 860 m+1-n after thefirst sequence of images 860 a-m. The keyframe manager 1120 can beconfigured to update the keyframe responsive to receiving each image ofthe sequence of images 860 m+1-n. Each of the keyframes generated by thekeyframe manager can be updated as each subsequent image of the sequenceof images is processed. In this way, as more and more images areprocessed, the keyframes are updated to include additional data, therebyimproving the accuracy and quality of the keyframe, and as a result, theaccuracy and quality of the 3D world map generated from the keyframesand corresponding point clouds.

In some implementations, the keyframe includes a first keyframe, and thekeyframe manager 1120 is configured to update the point cloud with asecond keyframe. The second keyframe can be generated using the firstkeyframe and a subset of the second sequence of images 860 m+1-ncaptured after the first keyframe is generated. In variousimplementations, the first sequence of images 860 a-m may or may not beconsecutively or directly followed by the second sequence of images 860m+1-n. For example, the 3D world map manager 815 may continuouslyreceive a stream of images in real-time. Various subsets of imageswithin the stream may be used for generating a given keyframe, such asto resolve movement issues or determine depth information that may notbe accessible or able to be determined from two consecutive sequences ofimages. In some implementations, a first sequence of images may capturea relatively static or time-invariant portion of the environment aboutthe movable entity (e.g., an object which may be relatively far from themovable entity or relatively large as compared to the movable entity,such as a mountain or other feature of the terrain), such that the firstsequence of images may be valuable for generating keyframes for severalsequences of images.

In some implementations, the depth value estimator 1115 is configured toestimate the depth value for pixels using motion data of the movableentity. For example, the depth value estimator 1115 can be configured toestimate the depth value for each pixel of the subset of the firstplurality of pixels using motion data for the movable entity such asvelocity or acceleration information. The depth value may be an absolutedepth (e.g., relative to a frame of reference which may be determinedprior to receiving the sequence of images 860 a-n or independent of thesequence of images 860 a-n, such as a frame of reference based on GPScoordinates), or a relative depth (e.g., relative to a frame ofreference which may be determined based on images captured by the imagecapture device 855 or otherwise related to the movable entity).

The keyframe manager 1120 can be configured to publish a depth mapincluding the estimated depth values for each pixel of the subset of thefirst plurality of pixels for the first image that are being trackedacross the sequence of images 860 a-n. The depth map can include, foreach tracked pixel, absolute distance information calculated based on adistance the tracked pixel moved across the sequence of images 860 a-nand based on the motion data (e.g., velocity) of the movable entity.

The 3D world map generator 1125 is configured to generate a 3D worldmodel. The 3D world map generator 1125 can generate the 3D world modelusing or based on the generated keyframe (e.g., generated by thekeyframe manager 1120) and/or the point cloud. The 3D world mapgenerator 1125 can maintain a representation of the 3D world model(e.g., a representation of expected features of the 3D world modeland/or the environment about the movable entity). The 3D world mapgenerator 1125 can compare the generated keyframe to the representationto determine if there are expected features of the 3D world model notpresent in the generated keyframe, and combine the expected featureswith the generated keyframe to generate the 3D world model.

The 3D world map generator 1125 can be configured to update the 3D worldmodel over time. For example, the 3D world map generator 1125 can use afirst keyframe and/or a first point cloud as a reference. As the 3Dworld map manager 815 receives additional sequences of images 860 a-nand generates additional keyframes, the 3D world map generator 1125 canadd the additional keyframes or associated point clouds (or features orpixels thereof) to the 3D world model.

In some implementations, the 3D world map generator 1125 selectivelyupdates the 3D world model. For example, the 3D world map generator 1125can compare one or more additional keyframes to the first keyframe,determine a difference based on the comparison, compare the differenceto a threshold, and include features or pixels of the one or moreadditional keyframes in the 3D world model if the difference is lessthan the threshold (e.g., if the comparison indicates that at least somefeatures or pixels of the one or more additional keyframes are similarto corresponding features or pixels of the first keyframe). In someimplementations, the 3D world map generator 1125 can be configured toweigh keyframes based on at least one of a time associated with thekeyframe (e.g., a time indicating recency of the keyframe) or motiondata associated with the keyframe, which can facilitate combininginformation from across keyframes based on similarities acrosskeyframes.

FIG. 12 is a flow chart depicting a method 1200 for generating a 3Dworld model, according to a non-limiting embodiment. It is to beemphasized, however, that method 1200 need not be performed in the exactsequence as shown, hence the elements of method 1200 are referred toherein as “blocks” rather than “steps.”

At block 1205, a sequence of images is received, such as by an imagereceiver. The sequence of images can be received from an image capturedevice mounted on a movable entity, such as image capture device 855.The sequence of images can include a first image captured at a firsttime and a plurality of second images captured after the first time. Forexample, the sequence of images may be associated with sequential timesor time stamps, such that the first image is captured at a first timeprior to when the plurality of second images are captured. The imagereceiver can be configured to determine the time associated with eachimage from the sequence of images, such as by extracting the time frommetadata transported with each image.

The first image of the sequence of images includes a first plurality ofpixels. The plurality of second images also each include a respectivesecond plurality of pixels. Each image can be represented as an orderedor organized set or plurality of pixels, such as an arrangement ofm-by-n pixels (e.g., m rows of pixels by n columns of pixels).

Each pixel (e.g., the first plurality of pixels, the respective secondplurality of pixels) has or is associated with one or more respectiveparameter values. The parameter values can store, encode, or otherwiserepresent image information captured in each image. For example, theparameter values may be associated with one or more of a color,intensity, luminance, or other parameter representing image informationor indicative of the light associated with the pixel captured in theimage. In some implementations, each pixel is associated with agreyscale intensity value and one or more values representing color(e.g., representing color in an RGB color format).

In some implementations, the pixels of the sequence of images areidentified. For example, the pixels can be extracted from the sequenceof images. A pixel identifier can receive an image and extract thepixels from the image, such as by storing parameter value information ina map or other data structure (such as a database) corresponding to eachimage of the sequence of images.

In some implementations, the pixel identifier selectively identifiespixels. For example, a filter can be applied to the sequence of imagesto selectively identify pixels. The filter can be based on one or moreparameter values. For example, the filter can be based on an intensityof each pixel.

At block 1210, motion data of a movable entity is received. For example,a motion data receiver or tracker can receive the motion data. Themotion data can include velocity information and/or accelerationinformation regarding the movable entity. The motion data can alsoinclude attitude data or other data representing a motion state orenergy state of the movable entity (e.g., attitude, thrust, heading,bearing, orientation). The motion data can also include or represent aposition of the movable entity (e.g., an absolute position, such as GPScoordinates; a relative position such as a position relative to a leaderor follower object or a base station).

Each image of the sequence of images can be associated with a respectivemotion data value of the movable entity corresponding to a time theimage was captured by the image capture device. As such, the dynamicinformation regarding the movable entity can be generated and stored,such as a profile representing position, velocity, or other motioninformation over time. The profile can be used to determine depthinformation for each image or otherwise generate a model of theenvironment about the movable entity

At block 1215, movement of pixels in or across the sequence of images istracked. Movement of at least a subset of the first plurality of pixelsof the first image across the sequence of images can be tracked. Forexample, as the movable entity changes in position and/or orientation,pixels may enter or exit the images captured by a fixed camera of themovable entity, such that only a subset of the first plurality of pixelsof the first image may be tracked across the sequence of images. Thepixels in the sequence of images may be associated with features in anenvironment about the movable entity which may be static (e.g., theabsolute positions of the features do not change over time, though thepixels associated with those features in the images may change as therelative position of the movable entity changes as compared to thefeatures) or dynamic (e.g., the absolute positions of the featureschange over time regardless of whether the absolute position of themovable entity changes). For example, given an m-by-n arrangement ofpixels, a feature (e.g., a pixel-sized feature as captured by the imagecapture device) of the environment may be associated with a pixel inposition [m₁, n₁] of the first image, and with a pixel in position[m₂,n₂] of the second image. Movement of the pixel (e.g., movement of afeature represented by the pixel) can be tracked from position [m₁, n₁]to position [m₂, n₂] from the first image to the second image.

In some implementations, pixel movement is based on or using parametervalues. For example, movement of at least the subset of the firstplurality of pixels of the first image can be tracked across thesequence of images using the respective parameter values of the subsetof the first plurality of pixels.

Movement can be tracked based on determining a match between a firstpixel of the first plurality of pixels and a second pixel of the secondplurality of pixels. For example, the first pixel can be identifiedbased on a first parameter value of the first pixel (e.g., a firstintensity). The second plurality of pixels can be searched to identifyor determine one or more second pixels having a second parameter value(e.g., a second intensity) corresponding to or matching the firstparameter value. In some implementations, a match or correspondence isdetermined or identified based on comparing the first parameter value tothe second parameter value. For example, a difference can be determinedbetween the first parameter value and the second parameter value, andcompared to a threshold value; if the difference is less than athreshold value, then a match or correspondence can be determined toexist between the first pixel and second pixel. In some implementations,an initial or preliminary match or correspondence can be determined, andfurther refined to identify a final match or correspondence based on atleast one of a greatest similarity (e.g., least difference) between thefirst pixel and second pixel or an expected location of the secondpixel.

At block 1220, a depth value for pixels of the first plurality of pixelsis estimated. In some implementations, the estimation is performedresponsive to the tracked subset of the first plurality of pixels andusing the received motion data, a depth value for each pixel of thesubset of the first plurality of pixels. The depth value can beestimated based determining an indication of movement from the motiondata. For example, a motion state profile over time (e.g., position overtime, velocity over time, attitude over time) of the movable entity canbe determined, such as by comparing absolute or relative velocities ofthe movable entity associated with the motion data. In someimplementations, as the magnitude of a change in the motion state of themovable entity increases between two points in time, a relativelygreater difference in depth between the images corresponding to the twopoints in time can be determined.

At 1225, a keyframe is generated to populate a point cloud. For example,using the estimated depth value for each pixel of the subset of thefirst plurality of pixels, a keyframe can be generated to populate thepoint cloud. The point cloud can include a plurality of points. Eachpoint can correspond to a respective pixel of the subset of the firstplurality of pixels. Where the image data (e.g., the subset of the firstplurality of pixels) may be a two-dimensional representation of theenvironment about the movable entity, the keyframe provides a 3Drepresentation by including the depth value for each pixel. For example,a transformation can be applied to the subset of the first plurality ofpixels to convert the subset of the first plurality of pixels from atwo-dimension representation (e.g., an m-by-n arrangement of pixelswhich may have corresponding parameter values such as intensity andcolor) to a three-dimension representation (e.g., an m-by-n-by-karrangement of pixels which may have the same corresponding parametervalues such as intensity and color).

In some implementations, the sequence of images includes a firstsequence of images. A second sequence of images 860 m+1-n can bereceived after the first sequence of images. The keyframe can be updatedresponsive to receiving each image of the sequence of images.

In some implementations, the keyframe includes a first keyframe, and thepoint cloud can be updated with a second keyframe. The second keyframecan be generated using the first keyframe and a subset of the secondsequence of images captured after the first keyframe is generated. Invarious implementations, the first sequence of images may or may not beconsecutively or directly followed by the second sequence of images. Forexample, a stream of images may be continuously received in real-time.Various subsets of images within the stream may be used for generating agiven keyframe, such as to resolve movement issues or determine depthinformation that may not be accessible or able to be determined from twoconsecutive sequences of images. In some implementations, a firstsequence of images may capture a relatively static or time-invariantportion of the environment about the movable entity (e.g., an objectwhich may be relatively far from the movable entity or relatively largeas compared to the movable entity, such as a mountain or other featureof the terrain), such that the first sequence of images may be valuablefor generating keyframes for several sequences of images.

In some implementations, the depth value for pixels can be estimatedusing motion data of the movable entity. For example, the depth valuefor each pixel of the subset of the first plurality of pixels can beestimated using motion data for the movable entity such as velocity oracceleration information. The depth value may be an absolute depth(e.g., relative to a frame of reference which may be determined prior toreceiving the sequence of images or independent of the sequence ofimages, such as a frame of reference based on GPS coordinates), or arelative depth (e.g., relative to a frame of reference which may bedetermined based on images captured by the image capture device 855 orotherwise related to the movable entity).

A depth map including the estimated depth values for each pixel of thesubset of the first plurality of pixels for the first image that arebeing tracked across the sequence of images can be generated orpublished. The depth map can include, for each tracked pixel, absolutedistance information calculated based on a distance the tracked pixelmoved across the sequence of images and based on the motion data (e.g.,velocity) of the movable entity.

At 1230, a 3D world model is generated. The 3D world model can begenerated using or based on the generated keyframe and/or the pointcloud. A representation of the 3D world model (e.g., a representation ofexpected features of the 3D world model and/or the environment about themovable entity) can be retrieved. The generated keyframe can be comparedto the representation to determine if there are expected features of the3D world model not present in the generated keyframe, and the expectedfeatures combined with the generated keyframe to generate the 3D worldmodel.

The 3D world model can be updated over time. For example, a firstkeyframe and/or a first point cloud can be used as a reference. Asadditional sequences of images are received and additional keyframes aregenerated, the additional keyframes or associated point clouds (orfeatures or pixels thereof) can be added to the 3D world model.

In some implementations, the 3D world model is selectively updated. Forexample, one or more additional keyframes can be compared to the firstkeyframe, a difference determined based on the comparison, thedifference compared to a threshold, and features or pixels of the one ormore additional keyframes included in the 3D world model if thedifference is less than the threshold (e.g., if the comparison indicatesthat at least some features or pixels of the one or more additionalkeyframes are similar to corresponding features or pixels of the firstkeyframe). In some implementations, keyframes and their associatedindividual depth or pixel data can be weighed based on at least one of atime associated with the keyframe (e.g., a time indicating recency ofthe keyframe) or motion data associated with the keyframe, which canfacilitate combining information from across keyframes based onsimilarities across keyframes. In some implementations, the positiondata of the pixels and/or the confidence level assigned to each of thepixels can be used to update the keyframes or to perform comparisonsacross keyframes.

E. Systems and Methods for Dynamic Object Tracking

FIG. 13 is a flowchart depicting a method 1300 for updating a dynamicobject tracking model, according to a non-limiting embodiment. It is tobe emphasized, however, that method 1300 need not be performed in theexact sequence as shown, hence the elements of method 1300 are referredto herein as “blocks” rather than “steps”.

At block 1302, an image is captured. In some implementations, the imagecan be captured by the camera 104 on the device 100.

At block 1304, a dynamic object tracking model is initialized.Initialization of a dynamic object tracking model can be similar toupdating a situational awareness model, as described at block 1314, withthe added difference that the situational awareness model is beinggenerated with a limited number of images. In some implementations, thesituational awareness model is generated based on a completely randominitialization process or a combination of random and non-randomalgorithmically selected images. The added difference could includealgorithmic shortcuts in which confidence is temporarily decreased inorder to create a full map in a small amount of steps.

At block 1306, a dynamic object tracking model is stored as a sequenceof objects currently being tracked with their own parameters and storedinformation. The objects can be assigned a confidence level, which isalso stored in the dynamic object tracking model. The stored dynamicobject tracking model can be used in the generation or population of asparse point cloud.

At block 1308, the next image is captured and is fed into the remainingblocks in the method.

At blocks 1310 and 1312, a blob filter can be used to detect anomaliesin the environment and segment possible moving targets. Here, a numberof sets of pixels (“blobs”) can be identified. The contrast of thecontents of the “blob” as compared to its surrounding can be used tocreate a contrast ratio quantity to be able to discard objects which arenot real threats and simply background noise. Each segmented “blob” isgiven a certainty number based on the confidence measurement andcontrast ratio and is tracked in order to determine velocity, directionand distance from the system. Where there is greater contrast, there maybe a greater likelihood that the set of pixels is representing a dynamicobject. In some implementations, other types of filters may also beemployed. For instance, corner detection filters, feature detectionfilters and general interest point detection filters can be used todetect anomalies or features in the images, which can then be used toidentify dynamic objects as well as static objects.

At block 1314, the dynamic object tracking model is updated withinformation obtained from the blob filter as well as one or more of thecorner detection filters, feature detection filters and general interestpoint detection filters described above. This action can require aposition matching step in which all currently tracked objects arematched to the new objects, propagating their states in order to updatethe model. The velocities and positions of these objects at the sametime can also be updated.

FIG. 14 is a block diagram depicting components of a dynamic objectidentifier 820 used in the real time visual situational awareness systemshown in FIG. 8, according to a non-limiting embodiment. The dynamicobject identifier 820 can include a point of interest identifier 1405, apoint of interest movement tracker 1410, and a dynamic object detector1415. The point of interest identifier 1405, point of interest movementtracker 1410, and dynamic objector detector 1415 can include or executeat least one computer program or at least one script, and can includecombinations of software and hardware, such as one or more processorsconfigured to execute one or more scripts.

The dynamic object identifier 820 is configured to receive images 860a-n. Similar to the 3D world map manager 815, the dynamic objectidentifier 820 can receive a sequence of images captured by an imagecapture device mounted on a movable entity such as the movable entity(e.g., image capture device 855). The sequence of images can include afirst image captured at a first time and a plurality of second imagescaptured after the first time. For example, the sequence of images maybe associated with sequential times or time stamps, such that the firstimage is captured at a first time prior to when the plurality of secondimages are captured. The dynamic object identifier 820 can be configuredto determine the time associated with each image from the sequence ofimages, such as by extracting the time from metadata transported witheach image. As described with reference to the 3D world map manager 815,a first image 860 a can include a plurality of pixels having orassociated with one or more respective parameter values; the sequence ofimages 860 a-n can also include a plurality of second images 860 a-neach also including a respective second plurality of pixels that have orare associated with one or more respective parameter values. In someimplementations, the dynamic object identifier 820 is configured toidentify the first image 860 a of the sequence of images 860 a-n, suchas by identifying or extracting a time or time stamp associated witheach image to identify the first image 860 a. The first image 860 a maybe the image captured chronologically first.

The point of interest identifier 1405 is configured to identify a pointof interest in or on the first image 860 a. For example, the point ofinterest identifier 1405 can identify a point of interest that includesone or more first pixels of the first image 860 a. In someimplementations, the point of interest identifier 1405 is configured toidentify a plurality of points of interest on the first image. Eachpoint of interest of the plurality of points of interest can be mappedto a respective pixel of the first plurality of pixels.

In some implementations, the point of interest identifier 1405 isconfigured to identify a point of interest using one or more imageanalysis techniques. For example, the point of interest identifier 1405can determine a contrast ratio between pixels to identify pixels thatare distinct relative to surrounding pixels. The point of interestidentifier 1405 can execute an edge search, feature trace, or featuresearch algorithm to identify points of interest. The point of interestidentifier 1405 can execute a blob detection or corner detectionalgorithm. The point of interest identifier 1405 can cluster groups ofsimilar pixels (e.g., pixels having similar parameter values) which maybe dissimilar to adjacent or surrounding groups of pixels to identify apoint of interest.

The dynamic object identifier 820 can determine parameter values for thepixels to which points of interest are mapped. For example, the dynamicobject identifier 820 can extract parameter values (e.g., intensity,luminance, color values) from the image data of the images 860 a-nassociated with particular pixels.

The point of interest identifier 1405 is configured to identify one ormore pixels in one or more of the plurality of second images 860 b-n(e.g., across the remaining images of the sequence of images 860 a-n)that have parameter values that are statistically similar to thedetermined parameter values. The identified pixels included in theplurality of second images 860 b-n can represent the points of interestidentified on the first image 860 a.

In some implementations, the point of interest identifier 1405 isconfigured to determine that parameter values are similar based onexecuting a similarity comparison algorithm. For example, the point ofinterest identifier 1405 can compare parameter values for two pixels todetermine a difference between the parameter values, and compare thedifference to a threshold value to determine if the pixels have similarparameters values (e.g., pixels having similar parameter values mayrepresent the same point of interest). In some implementations, thethreshold value is a predetermined value. In some implementations, thethreshold value can be a value that is based on how similar other pixelsin the image are to one another. In some implementations, the thresholdvalue can be a value that is based on how similar pixels in the imagewithin a predetermined distance from the point of interest pixel are toone another. In some implementations, the point of interest identifier1405 can be configured to execute a population comparison algorithm(e.g., a t-test), to determine whether a first plurality of pixelsassociated with a point of interest in a first image is similar to asecond plurality of pixels associated with a point of interest in asecond image. In some implementations, the parameters associated withthe point of interest identifier 1405 are used to match points ofinterest in the second image. The point of interest identifier 1405 mayutilize certain parameters or parameter values to identify certainpoints of interests. The point of interest identifier 1405 can then usethese parameters or parameter values to identify or match points ofinterests identified in the first image with points of interests insubsequent images, such as the second image. In some implementations,parameter values may be statistically similar if the values are within acertain threshold level of the parameter value of the pixel previouslydetermined from one or more preceding images. In some implementations,the parameter values may be statistically similar if the values are lessthan at least 10%, 15%, 20% or, 25% of the parameter value of the pixelpreviously determined from one or more preceding images.

The point of interest movement tracker 1410 is configured to trackmovement of each point of interest across the plurality of second images860 b-n. The point of interest movement tracker 1410 can track movementof each point of interest by determining locations of the identifiedpixels having the parameter values statistically similar to the pixel towhich the point of interest is mapped in the plurality of second images860 a-n.

In some implementations, the point of interest movement tracker 1410 isconfigured to track movement of each point of interest based on anexpected motion profile, a trajectory, trended motion profile, or anextrapolated motion profile. For example, if two or more pixels in asecond image 860 m have parameter values that are statistically similarto the parameter value of a first pixel in the first image 860 a, thepoint of interest movement tracker 1410 may determine that it is morelikely that tracked movement is associated with one of the two or morepixels if the one pixel more closely aligns with the expected motionprofile. In some implementations, the use of an expected motion profilecan be applied across multiple second images 860 m (or across at leastthe first image 860 a and one or more second images 860 m) to determinetrends of pixel movement.

The dynamic object identifier 820 is configured to determine aconfidence score of the point of interest indicating a likelihood thatthe point of interest represents a dynamic object. For example, thedynamic object identifier 820 can compare the tracked movement of thepoint of interest to a model or representation of object movement. Themodel or representation of object movement may indicate an expectedmotion profile, such as a straight-line or smooth curve profile, or amotion profile having acceleration and/or deceleration phases consistentwith object movement (e.g., it may be less likely that tracked movementof pixels indicate a dynamic object if the tracked movement suggestsacceleration or deceleration of several Gs, or suggests discontinuous orrandom motion).

In some implementations, the dynamic object identifier 820 is configuredto determine the confidence score of the point of interest by comparing,for each image of at least a subset of the remaining images (e.g., theplurality of second images 860 b-n), a distance between a first locationwithin the image at which the pixel representing the point of interestwas expected to be located and a second location within the image atwhich the pixel representing the point of interest was located.

In some implementations, the dynamic object identifier 820 is configuredto determine that a point of interest represents a dynamic object basedon comparing that the distance between the first location and the secondlocation exceeds a predetermined distance. For example, if the distanceis less than a predetermined distance, it may be more likely that thepoint of interest is static or stationary, if the distance is greaterthan a predetermined distance, it may be more likely that the point ofinterest is dynamic or otherwise represents a dynamic object. In someimplementations, the dynamic object identifier 820 can be configured tocompare the distance to a plurality of thresholds. For example, if thedistance is less than a first distance (e.g., a first maximum distanceassociated with stationary objects), then the point of interest can bedetermined to be more likely to be stationary; if the distance isgreater than a second distance greater than the first distance (e.g., asecond minimum distance associated with dynamic objects), then the pointof interest can be determined to be more likely to be moving; if thedistance is greater than the first distance and less than the seconddistance, then the confidence score may not be adjusted based on thedistance threshold evaluation.

In some implementations, the dynamic object identifier 820 is configuredto determine that a second point of interest is a static object. Forexample, the dynamic object identifier 820 can determine a confidencescore for the second point of interest, and determine that theconfidence score fails to satisfy a threshold value based on thedistance between the first location and second location of the secondpoint of interest repeatedly being less than a predetermined distance(e.g., based on distance comparisons between the first image 860 a andat least two of the second images 860 b-n).

The dynamic object identifier 820 is configured to determine that atleast one point of interest of the plurality of points of interestrepresents a dynamic object responsive to determining that theconfidence score satisfies a threshold value. The threshold value may bea predetermined value, or may be determined and updated over time, suchas by training the dynamic object identifier 820 to a training sethaving known dynamic objects.

The dynamic object identifier 820 is configured to identify, from atleast one image of the sequence of images, a plurality of pixels thatrepresent the dynamic object represented by the point of interest. Forexample, the dynamic object identifier 820 can use the tracked movementto identify the pixels in each image of the sequence of images 860 a-nassociated with the point of interest.

In some implementations, the dynamic object identifier 820 is configuredto identify additional points of interest in the remaining images (e.g.,images 860 b-n) of the sequence of images 860 a-n. The dynamic objectidentifier 820 can track movement of the additional points of interestacross the remaining images to identify additional dynamic objects inthe sequence of images. The dynamic object identifier 820 can beconfigured to apply a filter to the first image 860 a to identify,isolate, distinguish, or categorize a plurality of points of interest(e.g., a blob filter). The dynamic object identifier 820 can beconfigured to track a plurality of points of interest across thesequence of images 860 a-n.

In some implementations, the dynamic object identifier 820 is configuredto receive motion data 870 (e.g., motion data 870 as described withreference to the 3D world map manager 815, such as by a motion datareceiver or tracker). The dynamic object identifier 820 can beconfigured to associate, to one or more images of the sequence of images860 a-n, a velocity (or other parameter associated with motion) at whichthe movable entity is moving at the time the image was captured.

The velocity information can facilitate determining or identifyingdynamic objects in the sequence of images 860 a-n by providing a frameof reference for the sequence of images 860 a-n (e.g., indicating abaseline for motion of pixels, features, and/or points of interestacross the sequence of images 860 a-n). For example, the dynamic objectidentifier 820 can determine an expected location for points of interest(or one or more pixels representing points of interest) in one or moreof the remaining images based on the motion data. The dynamic objectidentifier 820 can determine an expected location of an object usingcamera parameters, movement of the object relative to the camera and themovable entity, and a relative scaling of the bounding box that boundsthe object itself. The dynamic object identifier 820 can determine asize of the bounding box and use the determined size of the bounding boxto determine the size of the object. The dynamic object identifier 820can compare the expected location to an actual location at which thepoint of interest (or one or more pixels representing the point ofinterest) is located. Where the expected location and actual locationare relatively close (e.g., spaced by a distance less than a thresholddistance), it may be more likely that the point of interest represents astatic feature rather than a dynamic object, such that the confidencescore may be determined to be relatively low, or may be decreasedrelative to a determination of the confidence score that does notaccount for the motion data. Similarly, where the expected location andactual location are relatively far apart, it may be more likely that thepoint of interest represents a dynamic (or moving) object, such that theconfidence score may be determined to be relatively high, or may beincreased relative to a determination that does not account for themotion data.

In some implementations, the dynamic object identifier 820 is configuredto determine a shape of the dynamic object. For example, the dynamicobject identifier 820 can identify a subset of pixels adjacent to thepoint of interest representing the dynamic object. The dynamic objectidentifier 820 can identify the subset of pixels based on parametervalues of the subset of pixels, such as by identifying edges around thepoint of interest based on the parameter values. The dynamic objectidentifier 820 can also identify the parameter values corresponding tothe subset of pixels. The dynamic object identifier 820 (e.g., by thepoint of interest movement tracker 1410) can track movement of each ofthe subset of pixels across a second sequence of images received by thedynamic object identifier 820 after or subsequent to the first sequenceof images. The dynamic object identifier 820 can compare a displacementof the subset of pixels to movement of the pixel representing the pointof interest across the second sequence of images, such as to comparemotion profiles or tracked movement of the subset of pixels. Responsiveto determining that the displacement of the subset of pixels isstatistically similar to the movement of the pixel representing thepoint of interest (e.g., the subset of pixels follow a similar paththrough the images as the point of interest), the dynamic objectidentifier 820 can determine or identify that the subset of pixelsdefine a shape of the dynamic object.

In some implementations, as described above, once the dynamic objectidentifier 820 identifies a pixel identified as corresponding to thedynamic object, the dynamic object identifier 820 can analyze the pixelto identify one or more parameter values of the identified pixel. Thedynamic object identifier 820 can then identify adjacent pixels to theidentified pixel to determine one or more parameter values of theadjacent pixels. The dynamic object identifier 820 can then group pixelsadjacent to or proximate to the identified pixel representing the movingobject together based on common or similar parameter values. In someimplementations, the dynamic object identifier 820 can analyze theintensity values of these pixels to group them together. In someimplementations, the dynamic object identifier 820 can confirm thatthese pixels correspond to the movable entity by tracking their motionacross the sequence of images and determining that these pixels movesimilar to the identified pixel representing the moving object.

In some implementations, the dynamic object identifier 820 may identifypixels that also correspond to the dynamic object by identifying pixelsadjacent to or proximate to the identified pixel and identifying acontrast to other adjacent pixels. The dynamic object identifier 820 canthen determine a peak group of contrast values that represent the mostcontrast to the background or surrounding pixels. Those pixels that havea contrast value that is within a threshold value of the peak group ofcontrast values can be identified as belonging to or defining thedynamic object, while pixels having contrast values that are not withinthe threshold are determined to correspond to the background.

In some implementations, the dynamic object identifier 820 is configuredto determine a trajectory of the movable entity (for example, thevehicle) based on or using the motion data of the movable entity. Forexample, the dynamic object identifier 820 can use the motion data(e.g., position, velocity, and/or acceleration information) to calculatea trajectory of the movable entity, such as a series of pointsindicating positions of the movable entity over time (e.g., at timescorresponding to the sequence of images 860 a-n), as well as expectedfuture positions of the movable entity.

The dynamic object identifier 820 can be configured to determine acollision between the movable entity and the dynamic object defined bythe point of interest. For example, the dynamic object identifier 820can determine a trajectory of the point of interest based on the trackedmovement of the point of interest. The dynamic object identifier 820 canextrapolate a future location or path of the dynamic object defined bythe point of interest, and compare the future location or path to thetrajectory of the movable entity. The dynamic object identifier 820 candetermine that the movable entity and the dynamic object defined by thepoint of interest are on a collision path (e.g., determine anintersection between paths travelled by the objects to a degree ofconfidence).

In some implementations, the dynamic object identifier 820 is configuredto transmit a command indicating instructions to avoid a collision. Forexample, based on identifying the location of a collision, the dynamicobject identifier 820 can generate instructions indicating a change intrajectory of the movable entity that will result in collisionavoidance.

FIG. 15 is a flowchart depicting a method for updating a dynamic objecttracking model, according to a non-limiting embodiment. It is to beemphasized, however, that method 1500 need not be performed in the exactsequence as shown, hence the elements of method 1500 are referred toherein as “blocks” rather than “steps”.

At block 1505, a sequence of images captured by an image capture devicemounted on a movable entity is received. The sequence of images can becaptured by an image capture device mounted on a movable entity, such asthe vehicle. The sequence of images can include a first image capturedat a first time and a plurality of second images captured after thefirst time. For example, the sequence of images may be associated withsequential times or time stamps, such that the first image is capturedat a first time prior to when the plurality of second images arecaptured. The time associated with each image from the sequence ofimages can be determined, such as by extracting the time from metadatatransported with each image. As described with reference to the method1200, a first image can include a plurality of pixels having orassociated with one or more respective parameter values; the sequence ofimages can also include a plurality of second images each also includinga respective second plurality of pixels that have or are associated withone or more respective parameter values.

At block 1510, a first image of the sequence of images including a firstplurality of pixels is identified, the first image captured by the imagecapture device at a first time. The first image can be identified byidentifying or extracting a time or time stamp associated with eachimage to identify the first image. The first image may be the imagecaptured chronologically first.

At block 1515, using one or more image analysis techniques, a pluralityof points of interest on the first image can be identified. Each pointof interest of the plurality of points of interest can be mapped to arespective pixel of the first plurality of pixels. For example, thepoint of interest can be identified that includes one or more firstpixels of the first image. In some implementations, a plurality ofpoints of interest can be identified on the first image. Each point ofinterest of the plurality of points of interest can be mapped to arespective pixel of the first plurality of pixels of the first image.The image analysis techniques can include determining a contrast ratiobetween pixels to identify pixels that are distinct relative tosurrounding pixels. The image analysis techniques can include executingan edge search, feature trace, or feature search algorithm to identifypoints of interest. The image analysis techniques can include executinga blob detection or corner detection algorithm. The image analysistechniques can include clustering groups of similar pixels (e.g., pixelshaving similar parameter values) which may be dissimilar to adjacent orsurrounding groups of pixels to identify a point of interest.

At block 1520, parameter values for the pixels to which the plurality ofpoints of interest are mapped can be determined. For example, parametervalues (e.g., intensity, luminance, color values) can be extracted fromthe image data of the images associated with particular pixels.

At block 1525, across the remaining images of the sequence of images,using the determined parameter values for the pixels to which theplurality of points of interest identified on the first image aremapped, pixels included in the remaining images that have parametervalues that are statistically similar to the determined parameter valuescan be identified, the identified pixels included in the remainingimages representing the points of interest identified on the firstimage. The identified pixels included in the plurality of second imagescan represent the points of interest identified on the first image.

In some implementations, parameter values are determined to be similarbased on executing a similarity comparison algorithm. For example,parameter values for two pixels can be compared to determine adifference between the parameter values, and the difference compared toa threshold value to determine if the pixels have similar parametersvalues (e.g., pixels having similar parameter values may represent thesame point of interest). In some implementations, a populationcomparison algorithm (e.g., a t-test) can be executed to determinewhether a first plurality of pixels associated with a point of interestin a first image is similar to a second plurality of pixels associatedwith a point of interest in a second image.

At block 1530, across the remaining images of the sequence of images,for each identified point of interest, movement of the point of interestcan be tracked by determining locations of the identified pixels havingthe parameter values statistically similar to the pixel to which thepoint of interest is mapped in the remaining images. Movement of eachpoint of interest can be tracked by determining locations of theidentified pixels having the parameter values statistically similar tothe pixel to which the point of interest is mapped in the plurality ofsecond images.

In some implementations, movement of each point of interest can betracked based on an expected motion profile, a trajectory, trendedmotion profile, or an extrapolated motion profile. For example, if twoor more pixels in a second image have parameter values that arestatistically similar to the parameter value of a first pixel in thefirst image, it may be determined that it is more likely that trackedmovement is associated with one of the two or more pixels if the onepixel more closely aligns with the expected motion profile. In someimplementations, the use of an expected motion profile can be appliedacross multiple second images (or across at least the first image andone or more second images) to determine trends of pixel movement.

At block 1535, a confidence score of the point of interest indicating alikelihood that the point of interest represents a dynamic object isdetermined by comparing, for each image of at least a subset of theremaining images, a distance between a first location within the imageat which the pixel representing the point of interest was expected to belocated and a second location within the image at which the pixelrepresenting the point of interest was located. For example, the trackedmovement of the point of interest can be compared to a model orrepresentation of object movement. The model or representation of objectmovement may indicate an expected motion profile, such as astraight-line or smooth curve profile, or a motion profile havingacceleration and/or deceleration phases consistent with object movement(e.g., it may be less likely that tracked movement of pixels indicate adynamic object if the tracked movement suggests acceleration ordeceleration of several Gs, or suggests discontinuous or random motion).In some implementations, the distance can be represented by a number ofpixels. In some implementations, the distance can be represented by adistance the dynamic object appeared to travel. In some suchimplementations, the distance can be represented by a unit ofmeasurement, such as meters, feet, among others.

At block 1540, it is determined that at least one point of interest ofthe plurality of points of interest represents a dynamic objectresponsive to determining that the confidence score of the point ofinterest satisfies a threshold value. The threshold value may be apredetermined value, or may be determined and updated over time, such asby training the dynamic object identifier 820 to a training set havingknown dynamic objects. The confidence score can be determined based oncomparing a distance between an expected location of the point ofinterest and an actual location of the point of interest to a threshold.In some implementations, a point of interest represents a dynamic objectbased on comparing that the distance between the first location and thesecond location exceeds a predetermined distance. For example, if thedistance is less than a predetermined distance, it may be more likelythat the point of interest is static or stationary; if the distance isgreater than a predetermined distance, it may be more likely that thepoint of interest is dynamic or otherwise represents a dynamic object.In some implementations, the distance can be compared to a plurality ofthresholds. For example, if the distance is less than a first distance(e.g., a first maximum distance associated with stationary objects),then the point of interest can be determined to be more likely to bestationary; if the distance is greater than a second distance greaterthan the first distance (e.g., a second minimum distance associated withdynamic objects), then the point of interest can be determined to bemore likely to be moving; if the distance is greater than the firstdistance and less than the second distance, then the confidence scoremay not be adjusted based on the distance threshold evaluation. In someimplementations, a point of interest can be determined to be a staticobject based on multiple threshold comparisons, such as if theconfidence score fails to satisfy a threshold value based on thedistance between the first location and second location of the secondpoint of interest repeatedly being less than a predetermined distance(e.g., based on distance comparisons between the first image and atleast two of the second images).

At block 1545, it is identified, from at least one image of the sequenceof images, a plurality of pixels that represent the dynamic objectrepresented by the point of interest. For example, the tracked movementcan be used to identify the pixels in each image of the sequence ofimages associated with the point of interest.

In some implementations, additional points of interest can be identifiedin the remaining images of the sequence of images. Movement of theadditional points of interest can be tracked across the remaining imagesto identify additional dynamic objects in the sequence of images. Afilter can be applied to the first image to identify, isolate,distinguish, or categorize a plurality of points of interest (e.g., ablob filter). A plurality of points of interest can be tracked acrossthe sequence of images.

In some implementations, motion data can be received. A velocity (orother parameter associated with motion) at which the movable entity ismoving at the time the image was captured can be configured toassociated to one or more images of the sequence of images.

The velocity information can facilitate determining or identifyingdynamic objects in the sequence of images by providing a frame ofreference for the sequence of images (e.g., indicating a baseline formotion of pixels, features, and/or points of interest across thesequence of images). For example, an expected location for points ofinterest (or one or more pixels representing points of interest) can bedetermined or identified in one or more of the remaining images based onthe motion data. The expected location can be compared to an actuallocation at which the points of interest (or one or more pixelsrepresenting points of interest) are located. Where the expectedlocation and actual location are relatively close (e.g., spaced by adistance less than a threshold distance), it may be more likely that thepoint of interest represents a static feature rather than a dynamicobject, such that the confidence score may be determine to be relativelylow, or may be decreased relative to a determination of the confidencescore that does not account for the motion data. Similarly, where theexpected location and actual location are relatively far apart, it maybe more likely that the point of interest represents a dynamic object,such that the confidence score may be determined to be relatively high,or may be increased relative to a determination that does not accountfor the motion data.

In some implementations, a shape of the dynamic object can bedetermined. For example, a subset of pixels adjacent to the point ofinterest representing the dynamic object can be identified. The subsetof pixels can be identified based on parameter values of the subset ofpixels, such as by identifying edges around the point of interest basedon the parameter values. The parameter values corresponding to thesubset of pixels can also be identified. Movement of each of the subsetof pixels can be tracked across a second sequence of images receivedafter or subsequent to the first sequence of images. A displacement ofthe subset of pixels can be compared to movement of the pixelrepresenting the point of interest across the second sequence of images,such as to compare motion profiles or tracked movement of the subset ofpixels. Responsive to determining that the displacement of the subset ofpixels is statistically similar to the movement of the pixelrepresenting the point of interest (e.g., the subset of pixels follow asimilar path through the images as the point of interest), it can bedetermined or identified that the subset of pixels define a shape of themovable entity.

In some implementations, a trajectory of the movable entity can bedetermined based on or using the motion data of the movable entity. Forexample, the motion data (e.g., position, velocity, and/or accelerationinformation) can be used to calculate a trajectory of the movableentity, such as a series of points indicating positions of the movableentity over time (e.g., at times corresponding to the sequence ofimages), as well as expected future positions of the movable entity.

A collision can be determined between the movable entity and the dynamicobject defined by the point of interest. For example, a trajectory ofthe point of interest can be determined based on the tracked movement ofthe point of interest. A future location or path of the dynamic objectdefined by the point of interest can be extrapolated, and compare thefuture location or path to the trajectory of the movable entity. It canbe determined that the movable entity and the dynamic object defined bythe point of interest are on a collision path (e.g., determine anintersection between paths travelled by the objects to a degree ofconfidence).

In some implementations, a command is transmitted indicatinginstructions to avoid a collision. For example, based on identifying thelocation of a collision, instructions can be generated indicating achange in trajectory of the movable entity that will result in collisionavoidance.

F. Systems and Methods for Updating a Real-Time Situational AwarenessModel

FIG. 16 is a flowchart depicting a method 1600 for updating a real-timesituational awareness model, according to a non-limiting embodiment. Itis to be emphasized, however, that method 1600 need not be performed inthe exact sequence as shown, hence the elements of method 1600 arereferred to herein as “blocks” rather than “steps”.

FIG. 16 illustrates how the guidance system feeds into the real-timesituational awareness model. With each image collected, thephotogrammetric range imaging algorithms and dynamic tracking algorithmsfunction can combine to form a robust awareness of the environment.

At block 1602, an image is captured. In some implementations, the imagecan be captured by the camera 104 on the device 100.

At block 1604, unsupervised scene filtering is performed. It is throughunsupervised scene filter that the visual system can handle the vastnumber of indoor and outdoor environments through the intelligent scenehandling and detection system. This algorithm can use a set of featuressuch as, but not limited to, image noise, contrast, lighting, shadows,and saturation to be able to quantify scenes. The images can then begiven appropriate filtering based on the scene classification algorithm(e.g., contrast deviation in darker, cloudy scenes). The classificationparameters can come from an unsupervised machine learning algorithm thatis trained on thousands of sets of videos and reenactments of indoor andoutdoor flight scenes. This process can lead to a more robust, filteredscene, which is primed for analysis by the rest of the computer visionalgorithms employed in the system.

Traditionally, the external, unstructured environment is an issue forcomputer vision as light, contrast and the feature set are rarelyconsistent. The unsupervised algorithm used here is able todifferentiate scenes and normalize to improve computer vision results,creating better accuracy down the entire computer vision pipeline. Thispreprocessing algorithm adds value to the entire visual situationalawareness system by creating a better base from which to start creatingstructure and 3D depth maps.

At block 1606, the 3D world model is updated. This process is describedin greater detail with reference to FIGS. 10-12.

At block 1608, the Structure From Motion update occurs. This process isdescribed in greater detail with respect to FIG. 13.

At block 1610, an additional image is captured. This process isdescribed in greater detail at block 1012 of FIG. 10 and block 1308 ofFIG. 13.

At block 1612, a dynamic tracking model is updated. This process isdescribed in greater detail with reference to FIGS. 13-15.

At block 1614, a situational awareness model is updated. This process isdescribed in greater detail at block 906 of FIG. 9A and additionaldetails are provided herein with respect to FIGS. 17-18.

FIG. 17 is a block diagram depicting components of a situationalawareness manager used in the real time visual situational awarenesssystem shown in FIG. 8, according to a non-limiting embodiment.Situational awareness manager 825 can include a 3D world map receiver1705, a dynamic object receiver 1710, and a situational awareness modelgenerator 1715. The 3D world map receiver 1705, dynamic object receiver1710, and situational awareness model generator 1715 can include orexecute at least one computer program or at least one script, and caninclude combinations of software and hardware, such as one or moreprocessors configured to execute one or more scripts. Where thesituational awareness manager 825 is tasked with or configured toperform similar functions as described with reference to the 3D worldmap manager 815 or the dynamic object identifier 820, the situationalawareness manager 825 can include or be configured to execute similarroutines, modules, programs, or scripts as those other components.

Similar to the 3D world map manager 815 and dynamic object identifier820, the situational awareness manager 825 can receive or otherwiseaccess the sequence of images 860 a-n captured by an image capturedevice mounted on a movable entity such as the movable entity (e.g.,image capture device 855).

The situational awareness manager 825 is configured to identify motiondata of the movable entity, such as the vehicle. For example, thesituational awareness manager 825 can receive motion data 870 and/orextract motion data from control data (e.g., control signals forcontrolling operation and motion of the movable entity). In someimplementations, identifying the motion data includes receiving themotion data from the movable entity via a communication link.Identifying the motion data can include determining motion data via avelocity determination module that is executed using triangulation. Forexample, the velocity determination module can periodically determine aposition of the movable entity (e.g., GPS coordinates) and compare thepositions over time to determine the motion data.

The situational awareness manager 825 is configured to track movement ofat least a subset of pixels (e.g., of the first plurality of pixels) ofrespective images across the sequence of images. For example, thesituational awareness manager 825 can identify or extract parametervalues of the subset of pixels in a first image, and search for similarpixels in the remaining images based on the identified or extractedparameter values. The situational awareness manager 825 can executemovement tracking algorithms similar to those executed by the dynamicobject identifier 820.

The situational awareness manager 825 is configured to estimate depthvalues of the tracked subset of pixels. The situational awarenessmanager 825 can estimate the depth values responsive to tracking thesubset of the first plurality of pixels. The situational awarenessmanager 825 can estimate the depth values based on the motion data 870.For example, the situational awareness manager 825 can process themotion data 870 to determine position, velocity, and/or accelerationinformation, which can be used to transform locations of pixels in eachimage into three-dimensional coordinates associated with depth values.In some implementations, relatively greater depth values are associatedwith or assigned to pixels (or features represented by pixels) that havea relatively lesser difference or disparity between images (or betweendistances indicated by the motion across images), as a relatively farobject will tend to appear in a similar position or at a similar anglerelative to the image capture device 855 while the image capture device855 moves. The depth value may be an absolute depth value or a relativedepth value.

In some implementations, the depth value is estimated without loopclosure. For example, as compared to existing systems which may berequired to perform loop closure to recognize a previously-visitedlocation and/or establish a frame of reference, the situationalawareness manager 825 can be configured to estimate the depth valueindependent of loop closure by estimating the depth value based onmotion data. In addition, the depth value estimation is based on a pixelby pixel basis and independent of mapping the pixel to pixels of imagescaptured from a different image capture device.

In some implementations, the depth value is estimated based on adistance that a pixel moves across images. For example, for each pixelof the tracked subset of pixels, a distance the pixel moved across theimages can be estimated, such as by determining a difference betweencoordinates of the pixel and applying a multiplying factor to thedifference based on known spatial information regarding the environmentabout the movable entity, the resolution or range of the image capturedevice 855, and/or the motion data.

The situational awareness manager 825 is configured to generate a 3Dworld map based on and/or using the estimated depth values of pixels ofthe tracked subset. For example, the situational awareness manager 825can include routines similar to those included in the 3D world mapmanager 815, or can execute the 3D world map manager 815. Thesituational awareness manager 825 can retrieve a model of the 3D worldmap, and incorporate pixels of the tracked subset into the model basedon the depth values of the pixels. In some implementations, thesituational awareness manager 825 can use the 3D world map receiver 1705to retrieve 3D world map data 1130 from the 3D world map manager 815.

In some implementations, the situational awareness manager 825 isconfigured to identify a plurality of points of interest to trackdynamic objects. The plurality of points of interest can be identifiedfrom the sequence of images as being mapped to respective pixels of theplurality of pixels. The points of interest can be identified in asimilar manner as to the identification performed or executed by thepoint of interest identifier 1405 of the dynamic object identifier 820.For example, the points of interest can be identified by extracting theparameter values of the plurality of pixels and identifying pixelshaving parameter values that are distinct relative to at least someadjacent pixels. The situational awareness manager 825 can include orexecute routines similar to those of the dynamic object identifier 820to identify points of interest associated with dynamic objects. Thesituational awareness manager 825 can apply a contrast filter toidentify pixels based on a contrast ratio relative to pixels within apredetermined distance of the respective pixel (e.g., a relatively largecontrast ratio may indicate an object or other point of interest, or anedge thereof).

In some implementations, the situational awareness manager 825 can usethe dynamic object receiver 1710 to receive dynamic object data 1420from the dynamic object identifier 820. The situational awarenessmanager 825 can identify the pixels or groups of pixels associated witheach dynamic object as the dynamic object moves through the images 860a-n.

In some implementations, the situational awareness manager 825 can trackmovement of each point of interest across the plurality of images 860a-n. For example, the situational awareness manager 825 can determine oridentify a trajectory or other motion profile of the point of interest.The situational awareness manager 825 can estimate parameters of themotion profile such as velocity.

The situational awareness manager 825 is configured to identify one ormore dynamic objects from the tracked movements of the points ofinterest. For example, the situational awareness manager 825 can compareactual positions of the respective points of interest to expectedpositions of the respective points of interest. The expected positionsof the respective points of interest can be determined based on knownmotion data regarding the movable entity. For example, given atrajectory of the movable entity (which may be determined in a similarmanner as described with reference to the dynamic object identifier),the situational awareness manager 825 can estimate expected locationsfor points of interest across the plurality of images (e.g., if thepoints of interest are fixed, then any change in location of the pointof interest should correspond to the motion or trajectory of the movableentity). If a difference between the expected location and actuallocation is less than a threshold distance, it may be more likely thatthe point of interest corresponds to a static object; if the differenceis greater than a threshold distance (which may be greater than thedistance used to determine that the point of interest is a staticobject), then it may be more likely that the point of interestcorresponds to a dynamic object.

The situational awareness model generator 1715 is configured to generatea visual situational awareness model using the generated 3D world map(e.g., the 3D world map data or a world map generated based on the 3Dworld map data) and one or more dynamic objects. For example, thesituational awareness model generator 1715 can use position information(including depth values) associated with the points of interest of thedynamic objects to add the dynamic objects to the 3D world map. Thesituational awareness model generator 1715 can add distinguishingfeatures to the dynamic objects (e.g., trend or trajectory lines,highlights) to visually highlight the dynamic objects in the 3D worldmap, which can increase situational awareness for a user viewing the 3Dworld map. In some implementations, the 3D world map includes aplurality of points that correspond to respective pixels. The points canbe associated with or correspond to an estimate depth value.

FIG. 18 is a flowchart depicting a method 1800 for updating a real-timesituational awareness model, according to a non-limiting embodiment. Itis to be emphasized, however, that method 1800 need not be performed inthe exact sequence as shown, hence the elements of method 1800 arereferred to herein as “blocks” rather than “steps”.

At block 1805, a sequence of images captured over time is received froman image capture device mounted on a movable entity. The sequence ofimages can be received from an image capture device mounted on a movableentity.

At block 1810, motion data of the movable entity is identified ordetected. Motion data can be received and/or extracted from control data(e.g., control signals for controlling operation and motion of themovable entity). In some implementations, identifying the motion dataincludes receiving the motion data from the movable entity via acommunication link. Identifying the motion data can include determiningmotion data via a velocity determination module that is executed usingtriangulation. For example, the velocity determination module canperiodically determine a position of the movable entity (e.g., GPScoordinates) and compare the positions over time to determine the motiondata.

At block 1815, movement of at least a subset of the plurality of pixelsof respective images is tracked across the sequence of images. Parametervalues of the subset of pixels in a first image can be identified orextracted, and similar pixels in the remaining images can be searchedfor based on the identified or extracted parameter values. Movementtracking algorithms similar to those executed by the dynamic objectidentifier 820 can be executed.

At block 1820, depth values of the tracked subset of pixels areestimated. The depth values can be estimated responsive to tracking thesubset of the first plurality of pixels. The depth values can beestimated based on the motion data. For example, the motion data can beprocessed to determine position, velocity, and/or accelerationinformation, which can be used to transform locations of pixels in eachimage into three-dimensional coordinates associated with depth values.In some implementations, relatively greater depth values are associatedwith or assigned to pixels (or features represented by pixels) that havea relatively lesser difference or disparity between images (or betweendistances indicated by the motion across images), as a relatively farobject will tend to appear in a similar position or at a similar anglerelative to the image capture device while the image capture devicemoves. The depth value may be an absolute depth value or a relativedepth value.

In some implementations, the depth value is estimated without loopclosure. For example, as compared to existing systems which may berequired to perform loop closure to recognize a previously-visitedlocation and/or establish a frame of reference, the depth value can bedetermined or estimated independent of loop closure by estimating thedepth value based on motion data.

In some implementations, the depth value is estimated based on adistance that a pixel moves across images. For example, for each pixelof the tracked subset of pixels, a distance the pixel moved across theimages can be estimated, such as by determining a difference betweencoordinates of the pixel and applying a multiplying factor to thedifference based on known spatial information regarding the environmentabout the movable entity, the resolution or range of the image capturedevice, and/or the motion data.

At block 1825, a 3D world map can be generated using the estimated depthvalues for the tracked subset. A model of the 3D world map can beretrieved, and can have pixels of the tracked subset incorporated intothe model based on the depth values of the pixels.

At block 1830, a plurality of points of interest mapped to respectivepixels of the plurality of pixels are identified to track dynamicobjects. The plurality of points of interest can be identified from thesequence of images as being mapped to respective pixels of the pluralityof pixels. The points of interest can be identified in a similar manneras to the identification performed in the methods 1200, 1500. Forexample, the points of interest can be identified by extracting theparameter values of the plurality of pixels and identifying pixelshaving parameter values that are distinct relative to at least someadjacent pixels. A contrast filter can be applied to identify pixelsbased on a contrast ratio relative to pixels within a predetermineddistance of the respective pixel (e.g., a relatively large contrastratio may indicate an object or other point of interest, or an edgethereof). In some implementations, pixels or groups of pixels associatedwith each dynamic object can be identified as the dynamic object movesthrough the images.

At block 1835, movement of each identified point of interest can betracked across the plurality of images. For example, a trajectory orother motion profile of the point of interest can be identified ordetermined. Parameters of the motion profile such as velocity can beestimated to facilitate movement tracking.

At block 1840, one or more dynamic objects are identified from thetracked movements of the points of interest. For example, actualpositions of the respective points of interest can be compared toexpected positions of the respective points of interest. The expectedpositions of the respective points of interest can be determined basedon known motion data regarding the movable entity. For example, given atrajectory of the movable entity (which may be determined in a similarmanner as described with reference to the dynamic object identifier),expected locations for points of interest across the plurality of imagescan be estimated (e.g., if the points of interest are fixed, then anychange in location of the point of interest should correspond to themotion or trajectory of the movable entity). If a difference between theexpected location and actual location is less than a threshold distance,it may be more likely that the point of interest corresponds to a staticobject; if the difference is greater than a threshold distance (whichmay be greater than the distance used to determine that the point ofinterest is a static object), then it may be more likely that the pointof interest corresponds to a dynamic object.

At 1845, a visual situational awareness model is generated using thegenerated 3D world map and the one or more dynamic objects. For example,position information (including depth values) associated with the pointsof interest of the dynamic objects can be used to add the dynamicobjects to the 3D world map. Distinguishing features can be added to thedynamic objects (e.g., trend or trajectory lines, highlights) tovisually highlight the dynamic objects in the 3D world map, which canincrease situational awareness for a user viewing the 3D world map. Insome implementations, the 3D world map includes a plurality of pointsthat correspond to respective pixels. The points can be associated withor correspond to an estimate depth value.

G. Systems and Methods for Trajectory Mapping

FIG. 19 is a block diagram depicting components of a trajectory manager830 used in the real time visual situational awareness system shown inFIG. 8, according to a non-limiting embodiment. The trajectory manager830 can include a dynamic object classifier 1805, dynamic objectclassification database 1910, dynamic object trajectory predictor 1915,and vehicle trajectory predictor 1920. The dynamic object classifier1805, dynamic object classification database 1910, dynamic objecttrajectory predictor 1915, and vehicle trajectory predictor 1920 caninclude or execute at least one computer program or at least one script,and can include combinations of software and hardware, such as one ormore processors configured to execute one or more scripts. Where thetrajectory manager 830 is tasked with or configured to perform similarfunctions as described with reference to the 3D world map manager 815,the dynamic object identifier 820, or the situational awareness manager825, the trajectory manager 830 can include or be configured to executesimilar routines, modules, programs, or scripts as those othercomponents. The trajectory manager 830 can receive the sequence ofimages 860 a-n, one or more dynamic objects from the dynamic objectidentifier 820, and motion data 870.

The trajectory manager 830 is configured to identify a pixelcorresponding to a dynamic object based on movement of the pixel acrossa plurality of images. For example, the trajectory manager 830 canselect a random pixel associated with the dynamic object, a distinctpixel (e.g., a pixel having a relatively great contrast ratio to otherpixels of the dynamic object or to other pixels outside the dynamicobject), or a pixel associated with an edge, outline, or border of thedynamic object. In some implementations, if the trajectory manager 830does not receive the dynamic object as a plurality of pixels, thetrajectory manager 830 can execute functions similar to those of thedynamic object identifier 820 to identify dynamic objects and pixelsassociated with dynamic objects.

The trajectory manager 830 is configured to identify, from the pixel, aplurality of pixels corresponding to the dynamic object. For example,the trajectory manager 830 compare parameter values of the identifiedpixel to parameter values of nearby pixels (e.g., within a thresholddistance, which may be associated with an expected size for a dynamicobject as informed by depth information) to identify similarities in thepixels or otherwise categorize the plurality of pixels as alsocorresponding to the dynamic object.

The trajectory manager 830 is configured to identify an outline definedby the identified plurality of pixels. For example, the trajectorymanager 830 can execute an image analysis algorithm to distinguish theplurality of pixels associated with the dynamic object from pixelssurrounding the dynamic object. The trajectory manager 830 can identifythe outline based on a contrast ratio between pixels defining theoutline and pixels outside the outline.

The dynamic object classifier 1905 is configured to determine visualcharacteristics of the dynamic object based on the outline defined bythe identified plurality of pixels. For example, the dynamic objectclassifier 1905 can extract or trace features of the outline, such asedges, corners, curves, or other visual characteristics, from theoutline. The dynamic object classifier 1905 can compare the extracted ortraced features to known features associated with dynamic objects. Forexample, the dynamic object classification database 1910 can include adatabase storing visual characteristics of dynamic objects. The dynamicobject classifier 1905 can retrieve the database of visualcharacteristics and compare the extracted or traced features to theretrieved characteristics to determine a match (e.g., a similarity-basedmatch).

The dynamic object classification database 1910 can store informationincluding types of dynamic objects, features associated with dynamicobjects, and mappings of features or other visual characteristics todynamic objects or types of dynamic objects. For example, the dynamicobject classification database 1910 can store visual characteristicsassociated with known features of dynamic objects such as engines,wings, cockpits, and rotors, as well as relative sizes of such features,and map object-specific features to types of objects.

The dynamic object classifier 1905 is configured to determine a type ofthe dynamic object according to the visual characteristics. For example,the dynamic object classifier 1905 can determine the type of dynamicobject based on determining that certain visual characteristicsassociated with or corresponding to certain type(s) of dynamic objectsare present in the outline or match known features of the types ofdynamic objects. The dynamic object classifier can determine the type ofdynamic object based on comparing visual characteristics to visualcharacteristics of known objects stored in the dynamic objectclassification database 1910. In some implementations, the dynamicobject classification database 1910 is configured to map visualcharacteristics to types of dynamic objects. The dynamic objectclassifier 1905 can process identified, extracted, or traced features orvisual characteristics of the dynamic object, select one or more visualcharacteristics in the dynamic object classification database 1910 thatare similar to the features or characteristics of the dynamic object,and retrieve the type of dynamic object associated with the selectedvisual characteristic.

The dynamic object trajectory predictor 1915 is configured to determinean absolute distance from the dynamic object. For example, the dynamicobject trajectory predictor 1915 can use a relative depth value ofpixels of the dynamic object, or compare an absolute depth value of thepixels of the dynamic object to an absolute location or depth value ofthe movable entity, to determine the absolute distance.

The dynamic object trajectory predictor 1915 is configured to generate,based on the type of the dynamic object and the absolute distance, apredicted trajectory of the dynamic object. For example, the dynamicobject trajectory predictor 1915 can determine a profile of movement ofthe dynamic object in the sequence of images, such as by fitting a curveto pixels associated with the dynamic object across the sequence ofimages. The dynamic object trajectory predictor 1915 can determine apredicted trajectory based on an expected motion or velocity of thedynamic object based on the type of the dynamic object. For example, thedynamic object trajectory predictor 1915 can identify an expectedvelocity or range of velocities associated with a type of dynamic object(e.g., a fixed wing aircraft, a rotary wing aircraft, a bird, a hot airballoon).

In some implementations, the dynamic object trajectory predictor 1915 isconfigured to update or scale at least one of dynamic object informationon a 3D world model based on motion data. For example, the dynamicobject trajectory predictor 1915 can use the motion data (e.g., GPSdata, velocity information) to determine an absolute distance traveledby the vehicle between a pair of images of the sequences of images 860a-n. Based on the absolute distance, the dynamic object trajectorypredictor 1915 can measure a length of features within each image, suchas a length of features or visual characteristics of dynamic objects, ora distance between dynamic objects.

The vehicle trajectory predictor 1920 is configured to generate, basedon the motion data, a trajectory of the movable entity, such as thevehicle. For example, the vehicle trajectory predictor 1920 can use acurrent velocity (e.g., speed and heading) of the movable entity topredict a future location of the movable entity. In someimplementations, the vehicle trajectory predictor 1920 is configured toextract a predicted maneuver from a plurality of past motion data (e.g.,a plurality of past positions and/or velocities). Based on the pluralityof past motion data, the vehicle trajectory predictor 1920 can estimatea maneuver being performed by the movable entity (e.g., a turn). Thevehicle trajectory predictor 1920 can predict the trajectory of themovable entity such that the movable entity follows a path expected tocomplete the predicted maneuver.

In some implementations, the dynamic object classification database 1910is configured to store a plurality of known maneuvers associated withtypes of dynamic objects, and the vehicle trajectory predictor 1920 canperform a lookup in the dynamic object classification database 1910 toidentify known maneuvers associated with the movable entity, compare theplurality of past motion data to the known maneuvers, and determine amatch in order to determine a predicted maneuver. In someimplementations, certain types of dynamic objects may perform specificmaneuvers that may be mandated or recommended by regulatory bodies orassociated with rules of the environment in which the vehicle isexpected to be operating.

In some implementations, the trajectory manager 830 is configured toestimate a collision based on the predicted trajectory of the dynamicobject and the trajectory of the movable entity. For example, thetrajectory manager 830 can determine whether an intersection will occur(e.g., to a degree of certainty) between the predicted trajectory of thedynamic object and the trajectory of the movable entity. The trajectorymanager 830 can generate and transmit an alert configured to indicatethe potential collision and/or a control instruction configured to alterthe trajectory of the movable entity to avoid the collision.

FIG. 20 is a flowchart depicting a method for generating a predictedtrajectory of an object, according to a non-limiting embodiment. It isto be emphasized, however, that method 1800 need not be performed in theexact sequence as shown, hence the elements of method 1800 are referredto herein as “blocks” rather than “steps”.

At block 2005, a pixel corresponding to a dynamic object is identifiedbased on a movement of the pixel across a plurality of images. A randompixel associated with the dynamic object, a distinct pixel (e.g., apixel having a relatively great contrast ratio to other pixels of thedynamic object or to other pixels outside the dynamic object), or apixel associated with an edge, outline, or border of the dynamic objectcan be selected. In some implementations, if the dynamic object is notreceived as a plurality of pixels, functions similar to those of methods1200, 1500 can be executed to identify dynamic objects and pixelsassociated with dynamic objects.

At block 2010, identify, from the pixel, a plurality of pixelscorresponding to the dynamic object. For example, parameter values ofthe identified pixel can be compared to parameter values of nearbypixels (e.g., within a threshold distance, which may be associated withan expected size for a dynamic object as informed by depth information)to identify similarities in the pixels or otherwise categorize theplurality of pixels as also corresponding to the dynamic object.

At block 2015, visual characteristics of the dynamic object aredetermined based on the outline defined by the identified plurality ofpixels. For example, an image analysis algorithm can be executed todistinguish the plurality of pixels associated with the dynamic objectfrom pixels surrounding the dynamic object. The outline can bedetermined based on a contrast ratio between pixels defining the outlineand pixels outside the outline.

At block 2020, a type of the dynamic object can be determined accordingto the visual characteristics. For example, features of the outline,such as edges, corners, curves, or other visual characteristics, can beextracted or traced from the outline. The extracted or traced featurescan be compared to known features associated with dynamic objects. Forexample, a database can be provided, generated, or stored having visualcharacteristics of dynamic objects. The database of visualcharacteristics can be retrieved and compare the extracted or tracedfeatures to the retrieved characteristics to determine a match (e.g., asimilarity-based match). The database can store information includingtypes of dynamic objects, features associated with dynamic objects, andmappings of features or other visual characteristics to dynamic objectsor types of dynamic objects. For example, the database can store visualcharacteristics associated with known features of dynamic objects suchas engines, wings, cockpits, and rotors, as well as relative sizes ofsuch features, and map object-specific features to types of objects.

The type of dynamic object can be determined based on determining thatcertain visual characteristics associated with or corresponding tocertain type(s) of dynamic objects are present in the outline or matchknown features of the types of dynamic objects. The type of dynamicobject can be determined based on comparing visual characteristics tovisual characteristics of known objects stored in the database. In someimplementations, visual characteristics are mapped to types of dynamicobjects. Identified, extracted, or traced features or visualcharacteristics of the dynamic object can be processed, one or morevisual characteristics in the database can be selected that are similarto the features or characteristics of the dynamic object, and the typeof dynamic object associated with the selected visual characteristic canbe retrieved.

At block 2025, an absolute distance from the dynamic object (vehicle)can be determined. For example, a relative depth value of pixels of thedynamic object can be used, or an absolute depth value of the pixels ofthe dynamic object can be compared to an absolute location or depthvalue of the movable entity, to determine the absolute distance. Forexample, a profile of movement of the dynamic object can be determinedin the sequence of images, such as by fitting a curve to pixelsassociated with the dynamic object across the sequence of images. Apredicted trajectory can be determined based on an expected motion orvelocity of the dynamic object based on the type of the dynamic object.For example, an expected velocity or range of velocities can bedetermined that are associated with a type of dynamic object (e.g., afixed wing aircraft, a rotary wing aircraft, a bird, a hot air balloon).

In some implementations, at least one of dynamic object information or a3D world model based on motion data can be scaled or updated based onmotion data. For example, motion data (e.g., GPS data, velocityinformation) can be used to determine an absolute distance between apair of images of the sequences of images. Based on the absolutedistance, a length of features within each image, such as a length offeatures or visual characteristics of dynamic objects, or a distancebetween dynamic objects, can be measured.

At block 2030, a previous path of the dynamic object can be determinedbased on the movement of the pixel representing the dynamic objectacross the plurality of images. Once a pixel being tracked is identifiedas corresponding to a dynamic object, the visual situational awarenesssystem can determine a path of the dynamic object. The path of thedynamic object can be used to determine the future trajectory of thedynamic object. In some implementations, the path of the dynamic objectcan be determined by tracking the movement of the pixel corresponding tothe dynamic object across the stream of images to determine a speed atwhich the pixel is travelling and a direction the pixel is moving. Insome implementations, the path of the pixel can be used to determine adistance of the object from the movable entity, such as the vehicle.

At block 2035, a predicted trajectory of the dynamic object can bedetermined based on the type of the dynamic object, the absolutedistance and the previous path travelled by the dynamic object. Forexample, a current velocity (e.g., speed and heading) of the movableentity can be used to predict a future location of the movable entity.In some implementations, a predicted maneuver can be extracted from aplurality of past motion data (e.g., a plurality of past positionsand/or velocities). Based on the plurality of past motion data, amaneuver being performed by the movable entity can be predicted,estimated, or determined (e.g., a turn). The trajectory of the movableentity can be determined such that the movable entity follows a pathexpected to complete the predicted maneuver. In some implementations, aplurality of known maneuvers associated with types of dynamic objectscan be stored in a database, and a lookup can be performed in thedatabase to identify known maneuvers associated with the movable entity,compare the plurality of past motion data to the known maneuvers, anddetermine a match in order to determine a predicted maneuver. In someimplementations, the predicted trajectory of one or more dynamic objectscan be based on the previous locations of the dynamic objects in the 3Dworld map. In some implementations, the predicted trajectory can bebased on the movement of the one or more tracked pixels representingeach dynamic object across a plurality of previous images to determinethe path the dynamic object previously was on, which can be used to thenpredict the further trajectory of the dynamic object.

In some implementations, a collision can be predicted or estimated basedon the predicted trajectory of the dynamic object and the trajectory ofthe movable entity. For example, a determination can be made whether anintersection will occur (e.g., to a degree of certainty) between thepredicted trajectory of the dynamic object and the trajectory of themovable entity. An alert can be generated and transmitted to indicatethe potential collision and/or a control instruction configured to alterthe trajectory of the movable entity to avoid the collision.

H. Systems and Methods for Collision Avoidance

FIG. 21 is a flowchart depicting a method 2100 for making a collisionavoidance instruction to a control system, according to a non-limitingembodiment. It is to be emphasized, however, that method 2100 need notbe performed in the exact sequence as shown, hence the elements ofmethod 2100 are referred to herein as “blocks” rather than “steps”.

Referring also to the collision avoidance manager 835 of FIG. 8, atblock 2102, the collision avoidance manager 835 can identify, receive,calculate or otherwise determine the trajectory of the vehicle, the oneor more dynamic objects identified by the real-time visual situationalawareness system 800. In addition, the collision avoidance manager 835can identify locations of various static objects identified from theimages to generate the 3D world model. The collision avoidance manager835 or the trajectory manager 830 can calculate, for each of the dynamicobjects identified, a trajectory of the dynamic object. The trajectoryof the dynamic objects can be used to identify a location at which eachof the dynamic objects will be at any given time.

The collision avoidance manager 835 can further determine locations atwhich the vehicle will be within the 3D world map based on a currentvelocity and acceleration of the vehicle. In some implementations, thecollision avoidance manager 835 can determine the locations at which thevehicle will be at various points in time based on a preplanned routethe vehicle is programmed to follow.

The collision avoidance manager 835 can be configured to generateupdated 3D world maps that populate the position of the various dynamicobjects relative to the vehicle over a period of time based on thepredicted trajectories of dynamic objects. In addition, the location ofvarious static objects are also updated to reflect their positionrelative to the vehicle as the vehicle moves along its trajectory.

At block 2104, the collision avoidance manager 835 can determine whethera collision between the vehicle and an object is imminent. In someimplementations, the collision avoidance manager 835 can determinewhether a collision is imminent based on the relative positions of thevehicle and one or more of the static or dynamic objects based on thepixel positions these objects will move to at various points in time.

At block 2106, if the collision avoidance manager 835 determines thatthere is no collision imminent, no avoidance maneuver is instructed. Insome implementations, the collision avoidance manager 835 mayperiodically check to see if the trajectory of the vehicle isintersecting a trajectory of a dynamic object or if the trajectory ofthe vehicle is intersecting a static object. In some implementations,the collision avoidance manager 835 may compute, responsive todetermining that the trajectory of the vehicle is likely to collide withanother object if the vehicle continues to follow its trajectory, adistance from the obstacle (dynamic object or static object with whichit is going to have a collision with).

The collision avoidance manager 835 can determine an absolute distancefrom the object or the projected path of a dynamic object based on the3D model, which is based on the velocity at which the vehicle istravelling. In some implementations, the collision avoidance manager 835may determine that the distance from the obstacle exceeds a minimumthreshold and as a result, may determine to not perform an avoidancemaneuver at that instant.

In some implementations, the collision avoidance manager 835 may furtherbe determined to compute a predicted change in the distance from theobstacle over a time period based on the trajectory of the vehicle aswell as the trajectory of the obstacle and compute a length of time,based on the current and future speed of the vehicle, from the instantafter which the collision will occur if no action is taken.

The collision avoidance manager 835 can determine, based on the lengthof time, whether to execute an avoidance maneuver. If the collisionavoidance manager determines that the length of time is greater than athreshold amount of time, the collision avoidance manager 835 can takeno avoidance maneuver (2106). However, if the collision avoidancemanager determines that the length of time is less than a thresholdamount of time, the collision avoidance manager 835 can generate aninstruction to perform a collision avoidance maneuver (2108). In someimplementations, the collision avoidance manager can be configured togenerate an alert to the vehicle or an operator of the vehicleindicating an imminent collision and can vary the intensity of the alertas the amount of time approaches the threshold amount of time. Once theamount of time reaches or is less than the threshold amount of time, thecollision avoidance manager 835 can generate an instruction to cause thevehicle to maneuver such that the vehicle and the obstacle are no longergoing to collide. The maneuver can include a change in direction ororientation, a change in speed, or both. In some implementations, thecollision avoidance maneuver can be determined in part with reference toa set of standard right-of-way rules. In some implementations, thecollision avoidance maneuver can be based on an alternate route of thevehicle. In some implementations, the collision avoidance maneuver canbe an instruction to land, loiter, circle or deploy a parachute (toreduce speed or adjust altitude, incline or decline), among others.

FIG. 26 is an image 2600 of an environment identifying points ofinterest in a terrain and a target indicator indicating a locationtowards which the movable entity is travelling, according to anon-limiting embodiment. The real-time visual situational awarenesssystem 800 can recognize objects or scenery from the image and provide avisual indicator indicating the terrain in the environment. Thereal-time visual situational awareness system 800 can also provide avisual indicator indicating a position to which the movable entity isapproaching. The size of the visual indicator can represent a potentialcollision object that the vehicle is potentially going to collide withif it continues on the same trajectory. The size of visual indicator bebased on a distance from the potential collision object.

FIG. 27A is an image 2700A of an environment captured by a camera of avehicle captured at a first time from a first position. FIG. 27B is animage 2700B of the same scene captured at a second time subsequent tothe first time from the same camera from a new position of the camera ofthe vehicle based on the path travelled by the first image. Thereal-time situational awareness system 800 can provide, for display, avisual indicator 2710 indicating a path the vehicle is heading in. Thevisual indicator can represent an area within which objects entering thearea may be identified or classified as potential collision objects. Asthe vehicle gets closer to the potential collision object within thearea represented by the visual indicator, the visual indicator maychange color, size, or shape to indicate that there is a collisionlikely to occur. In some implementations, the area defined by the visualindicator can be a collision avoidance zone indicator indicating a zonewithin which dynamic objects will be classified as potential threats. Asshown in FIG. 2700B, the dynamic object 2720 is a potential threat andthe real-time visual situational awareness system 800 may provide avisual indicator 2712 representing the potential threat. As the visualindicator 2712 enters the collision avoidance zone, the real-time visualsituational awareness system 800 may generate an alert or notificationindicating a likely collision.

FIGS. 28A and 28B are images 2800A-B of an environment including adynamic collision avoidance zone indicator indicating a zone withinwhich dynamic objects will be classified as potential threats. In FIG.28A, the vehicle is making a turn and as a result, the collisionavoidance zone indicator 2710 is extending from right to left on theimage corresponding to the direction of travel of the vehicle. Thecollision avoidance zone indicator can be based on control signalsreceived from a control system of the vehicle indicating a direction thevehicle is heading towards. In FIG. 28A, a dynamic object 2720 isoutside the central area of the collision avoidance zone indicator 2710.Referring now to FIG. 28B, the image 2800B is taken subsequent to image2800A and the collision avoidance zone indicator is no longer extendingfrom right to left on the image as the vehicle is now travellingstraight.

FIGS. 29A and 29B are images 2900A-B of an environment including astatic object identifier that changes color as the trajectory of themovable entity approaches the static object, according to a non-limitingembodiment. In FIG. 29A, the vehicle is at a first distance away fromthe object 2910. In FIG. 29B, the vehicle is now closer to the object(as the object appears larger) relative to FIG. 29B. The collisionavoidance manager can be configured to provide a visual indicator 2915on the portion of the object 2910 with which the vehicle will collide asthe vehicle reaches a certain distance from the object. The visualindicator can be lines of a certain color, but can be represented on theobject or adjacent to the object to draw the attention of a user to theobject. In some implementations, the indicator can be an auditoryindicator. In some implementations, the indicator can be a hapticindicator. In some implementations, the indicator can be a combinationof visual, audio and even haptic. IT should be appreciated that any ofthe indicators described herein can include any combination of visual,audio and haptic.

FIG. 30 is an image 3000 of an environment identifying a dynamic objectand a range indicator indicating a distance of the dynamic object fromthe movable entity, according to a non-limiting embodiment. Thereal-time visual situational awareness system 800 can be configured todisplay an indicator identifying a dynamic object 3010 identified by thedynamic object identifier and can also display a distance indicator 3020indicating a distance between the vehicle and the dynamic object 3010.Details of how the distance is calculated are provided throughout thedisclosure.

I. Systems and Methods for Mapping Control Signals to Vehicle MotionBased on Image Data

FIG. 22 is a block diagram depicting components of an control signalmotion mapper 840 used in the real time visual situational awarenesssystem shown in FIG. 8, according to a non-limiting embodiment. Thecontrol signal motion mapper 840 can include a vehicle motion statusmanager 2205, a motion status and control signal correlator 2210, avehicle command learning model generator 2215, and a vehicle controlinstruction generator 2220. The vehicle motion status manager 2205,motion status and control signal correlator 2210, vehicle commandlearning model generator 2215, and vehicle control instruction generator2220 can include or execute at least one computer program or at leastone script, and can include combinations of software and hardware, suchas one or more processors configured to execute one or more scripts.Where the control signal motion mapper 840 is tasked with or configuredto perform similar functions as described with reference to the 3D worldmap manager 815, the dynamic object identifier 820, the situationalawareness manager 825, the trajectory manager 830, or the collisionavoidance manager 835, the control signal motion mapper 840 can includeor be configured to execute similar routines, modules, programs, orscripts as those other components. The trajectory manager 830 canreceive the sequence of images 860 a-n, one or more dynamic objects fromthe dynamic object identifier 820, and motion data 870.

The vehicle motion status manager 2205 is configured to identify controldata sent to a vehicle drive system via an interface with the controlsystem of the vehicle. For example, the control signal motion mapper 840can receive a control signal input 875. The control signal input 875 canindicate control data sent to the vehicle drive system. The control datacan include or indicate instructions for controlling operation of thevehicle, such as autopilot or other motion or maneuvering instructions.In some implementations, the control data is represented by a softwaredriver in a motion controller language (e.g., hardware-specificinstructions), and the control signal motion mapper 840 is configured toparse the motion controller language to identify the control data. Insome implementations, the control data is represented at an applicationlevel by an application programming interface (e.g.,hardware-independent instructions), and the control signal motion mapper840 is configured to receive the control data via functions of theapplication programming interface exposed to the control signal motionmapper 840. In some implementations, the control signal motion mapper840 is configured to receive the control data as hardware-specificinstructions such as voltages required to cause targeted actions, andthe control signal motion mapper 840 is configured to apply a motioncontrol algorithm based on known properties of the control system of thevehicle to determine the original control instructions (e.g., detect a10 V control signal to the accelerator and determine that the controlinstructions indicated an acceleration of 80 percent maximumacceleration). In some implementations, the vehicle motion statusmanager 2205 is configured to identify a control scheme based on thecontrol signal input 875. For example, the vehicle motion status manager2205 can determine that the control signal input 875 indicates a pulsewidth modulation scheme or other scheme composed of a plurality ofelectrical signals, and record the control scheme.

In some implementations, the vehicle motion status manager 2205 isconfigured to identify the control data independent of the vehiclehardware. For example, the vehicle motion status manager 2205 may onlyreceive control signals in the form of hardware-specific instructionssuch as voltages to be applied to components, where the properties ofthose components are unknown. The vehicle motion status manager 2205 canstore the control signals as being mapped to particular components and,as will be described herein, estimate, interpolate, learn, or otherwisedetermine the associated vehicle actions and instructions based on adetected change in position and/or velocity of the vehicle withoutknowing the properties of the vehicle hardware (e.g., without knowing arelationship between the electrical signal of a control signal and theresponse of vehicle hardware such as engines, steering wheels, rotors,airfoils, wing control surfaces, etc.). This can enable the controlsignal motion mapper 840 to map hardware-specific commands to vehiclemaneuvers or other outcomes without being provided software drivers orother information indicating how to control the vehicle, which can allowapplication of the system in a “plug-and-play” manner even if thevehicle hardware information is proprietary or otherwise inaccessible.

The vehicle motion status and control signal correlator 2210 isconfigured to determine a change in position or velocity of the vehicle.For example, the vehicle motion status and control signal correlator2210 can analyze the motion data 870 to identify the change in positionor velocity. The vehicle motion status and control signal correlator2210 can also determine the change using or based on a 3D world mapgenerated from the sequence of images 860 a-n as described herein. Forexample, the vehicle motion status and control signal correlator 2210can identify motion of points of interest in the 3D world map (which arecaptured by the image capture device from the frame of reference of thevehicle) and determine the change in position or velocity based on thechange in frame of reference of the vehicle.

The vehicle motion status and control signal correlator 2210 can map thechange in position or velocity of the vehicle to the received controldata. For example, the vehicle motion status and control signalcorrelator 2210 can associate time points or time stamps of the changein position or velocity with time points or time stamps of the receivedcontrol data. In some implementations, the vehicle motion status andcontrol signal correlator 2210 is configured to apply a time delay tothe control data prior to associating the control data to the change inposition or velocity. For example, the vehicle motion status and controlsignal correlator 2210 can determine that a signal delay occurs betweena first point in time at which the control data is received and a secondpoint in time at which the change in position or velocity occurs. Insome implementations, the time delay is predetermined (e.g., bycalibrating the control signal motion mapper 840). In someimplementations, the vehicle motion status and control signal correlator2210 is configured to discretize the control data and associate thediscretized control data with the change in position or velocity. Forexample, the vehicle motion status and control signal correlator 2210can determine that the control data indicates an instruction over afirst duration of time (e.g., constantly accelerate at 50 percent ofmaximum acceleration for 3 seconds), that the change in position orvelocity is mapped over a second duration of time (e.g., the vehicle isat a first position at time zero seconds, a second position at time 0.1seconds, a third position at time 0.2 seconds, through an nth positionat time 3 seconds), and associate the acceleration at each time point tothe change in position or velocity.

In some implementations, the vehicle motion status and control signalcorrelator 2210 is configured to map the change in position or velocity(e.g., a known change in position or velocity) to the received controldata based on an expected maneuver. For example, the vehicle motionstatus and control signal correlator 2210 can execute a motion detectionalgorithm based on the received control data to estimate an expectedmaneuver (e.g., a control signal may be expected to indicateacceleration at 50 percent for 2 seconds), determine an expected changein position or velocity based on the expected maneuver (e.g., execute avehicle load algorithm to estimate an increase in velocity expectedbased on the acceleration at 50 percent for 2 seconds), and compare theexpected change in position or velocity to the known change in positionor velocity. Based on a difference between the expected change and theknown change, the vehicle motion status and control signal correlator2210 can calibrate the mapping to account for the difference.

The control signal motion mapper 840 (or another component of the system800) is configured to determine a condition to maneuver the vehicle. Forexample, the vehicle command learning model generator 2215 can receivean indication of a condition (e.g., receive an indication of analtitude, current speed, distance to destination, distance to a leaderor follower vehicle, distance to a dynamic object) and record theresulting maneuver performed by the vehicle based on the control dataand/or change in position or velocity. The conditions may be stored in adatabase and associated with maneuvers. In some implementations, theconditions may be stored and associated with types of dynamic objects.In some implementations, the conditions may be categorized according totypes of dynamic objects, types of conditions (e.g., known object inenvironment, unknown object in environment, position of vehicle relativeto environment). The conditions may be stored with a confidence levelindicating a confidence that a particular condition correlates to aparticular maneuver, which can facilitate decision-making for executingthe same maneuver to respond to the same condition.

In some implementations, the vehicle command learning model generator2215 is configured to map control instructions to a change in positionor velocity independent of the vehicle hardware. For example, thevehicle command learning model generator 2215 can receivehardware-specific control instructions (e.g., a voltage to be applied toa component, a control scheme including a plurality of electricalsignals such as a pulse width modulation scheme) and associate thehardware-specific control instructions with the change in position orvelocity. As such, the vehicle command learning model generator 2215 canlearn that a particular control scheme (e.g., apply a voltage of 10 Vfor 3 seconds to a first component) corresponds to a particular vehiclemaneuver (e.g., accelerate at 80 percent maximum acceleration for 3seconds) without knowing the underlying functionality of the vehiclehardware or the vehicle motion controller. In some implementations, thecontrol instructions are mapped and stored in a database. In someimplementation, the control instructions are associated with conditionsduring which the maneuvers (e.g., changes in position and/or velocity)occurred that took place at the same time as the control instructionswere received.

The vehicle control instruction generator 2220 is configured to generatecontrol instructions to maneuver the vehicle. For example, given amapping of control instructions to a change in position or velocity, thevehicle control instruction generator 2220 can generate a control signal(e.g., an electrical signal having a particular voltage, a pulse widthmodulation scheme) configured to cause a desired change in position orvelocity.

The vehicle control instruction generator 2220 can generate controlinstructions to maneuver the vehicle responsive to determining thecondition (e.g., determining that a dynamic object is on a collisioncourse with the vehicle). For example, the vehicle control instructiongenerator 2220 can retrieve control instructions associated with amaneuver that the system 800 learned to have taken place in response tothe same or a similar condition. The vehicle control instructiongenerator 2220 can retrieve control instructions from the variousdatabases described herein based on conditions (e.g., given a known orexpected condition, the vehicle control instruction generator 2220 canretrieve an expected maneuver to be used to respond to the condition).

The vehicle control instruction generator 2220 can transmit the controlinstructions to the control system of the vehicle via the interface. Forexample, the vehicle control instruction generator 2220 can output anelectrical signal corresponding to the control instructions via theinterface. In some implementations, the vehicle control instructiongenerator 2220 is configured to selectively output the controlinstructions based on a decision confidence. For example, whenretrieving the control instructions from a database, the vehicle controlinstruction generator 2220 can determine a confidence by which themaneuver indicated by the control instructions is expected to be anappropriate solution to the condition (e.g., based on stored confidencevalues when the maneuver is mapped to the condition in the database,such as confidence values regarding whether the condition existed and/orwhether the maneuver was performed responsive to the condition). Thevehicle control instruction generator 2220 can transmit the controlinstructions responsive to determining that the decision confidence isgreater than a threshold value. The threshold value can be apredetermined value. The threshold value can be adapted to the vehicleand/or the vehicle condition or a condition in an environment about thevehicle.

FIG. 23 is a flowchart for mapping control signals to vehicle motion,according to a non-limiting embodiment. It is to be emphasized, however,that method 2300 need not be performed in the exact sequence as shown,hence the elements of method 2300 are referred to herein as “blocks”rather than “steps”.

At block 2305, a stream or sequence of images is received by a computingdevice including one or more processors, from an image capture devicemounted on a vehicle. In some implementations, the images can becaptured by multiple cameras. In some implementations, each of thesequence images can be captured by multiple cameras and individualimages from the multiple cameras can be stitched or merged together toform the sequence image. It should be appreciated that other methodsdescribed herein with respect to FIGS. 9-23 may each utilize multiplecameras and merge or stitch images from the multiple cameras to formsingle images that together are the basis for the sequence of images.

At block 2310, control data sent to a vehicle drive system from acontrol system of the vehicle via an interface between the computingdevice and the control system of the vehicle is identified by thecomputing device. For example, a control signal input can be received.The control signal input can indicate control data sent to the vehicledrive system. The control data can include or indicate instructions forcontrolling operation of the vehicle, such as autopilot or other motionor maneuvering instructions. In some implementations, the control datais represented by a software driver in a motion controller language as ascript (e.g., hardware-specific instructions), and the script in themotion controller language is parsed to identify the control data. Insome implementations, the control data is represented at an applicationlevel by an application programming interface (e.g.,hardware-independent instructions), and the control data is received viafunctions of the application programming interface that are exposed tothe computing device. In some implementations, the control data isreceived as hardware-specific instructions such as voltages required tocause targeted actions, and a motion control algorithm is applied basedon known properties of the control system of the vehicle to determinethe original control instructions (e.g., detect a 10 V control signal tothe accelerator and determine that the control instructions indicated anacceleration of 80 percent maximum acceleration). In someimplementations, a control scheme is identified based on the controlsignal input. For example, the control signal input can be determined toindicate a pulse width modulation scheme or other scheme composed of aplurality of electrical signals, and record the control scheme.

In some implementations, the control data is identified independent ofthe vehicle hardware. For example, control signals may only be receivedin the form of hardware-specific instructions such as voltages to beapplied to components, where the properties of those components areunknown. The control signals can be stored as being mapped to particularcomponents and, as will be described herein, the computing device canestimate, interpolate, learn, or otherwise determine the associatedvehicle actions and instructions based on a detected change in positionand/or velocity of the vehicle without knowing the properties of thevehicle hardware (e.g., without knowing a relationship between theelectrical signal of a control signal and the response of vehiclehardware such as engines, steering wheels, rotors, airfoils, wingcontrol surfaces, etc.). This can enable the hardware-specific commandsto be mapped to vehicle maneuvers or other outcomes without beingprovided software drivers or other information indicating how to controlthe vehicle, which can allow a “plug-and-play” implementation even ifthe vehicle hardware information is proprietary or otherwiseinaccessible.

At block 2315, a change in a position or velocity of the vehicle using a3D world map generated from the received stream of images is determinedby the computing device. For example, the motion data can be analyzed toidentify the change in position or velocity. The change can also bedetermined using or based on a 3D world map generated from the sequenceof images as described herein. For example, motion of points of interestcan be identified in the 3D world map (which are captured by the imagecapture device from the frame of reference of the vehicle) and thechange in position or velocity determined based on the change in frameof reference of the vehicle.

At block 2320 the change in position or velocity of the vehicle can bemapped to the received control data. For example, the time points ortime stamps of the change in position or velocity can be associated withtime points or time stamps of the received control data. In someimplementations, a time delay can be applied to the control data priorto associating the control data to the change in position or velocity.For example, a signal delay can be determined to occur between a firstpoint in time at which the control data is received and a second pointin time at which the change in position or velocity occurs. In someimplementations, the time delay is predetermined (e.g., by calibratingthe computing device). In some implementations, the control data ismodified or discretized, and associated with the change in position orvelocity. For example, the control data can be determined to indicate aninstruction over a first duration of time (e.g., constantly accelerateat 50 percent of maximum acceleration for 3 seconds), that the change inposition or velocity is mapped over a second duration of time (e.g., thevehicle is at a first position at time zero seconds, a second positionat time 0.1 seconds, a third position at time 0.2 seconds, through annth position at time 3 seconds), and associate the acceleration at eachtime point to the change in position or velocity.

At block 2325, static objects and dynamic objects that appear within oneor more images of the stream of images can be identified and thepredicted trajectories of dynamic objects can be determined. Using thedynamic object identifier of the visual situational awareness system,the visual situational awareness system can identify one or more pixelsthat correspond to dynamic objects based on the movement of the pixelsacross the stream of images. Details of dynamic object detection areprovided above. Once a pixel corresponding to a dynamic object isidentified, the type of dynamic object is determined and a predictedtrajectory of the dynamic object is determined based on the movement ofthe pixel across the stream of images. In some implementations, thevisual situational awareness system determines this for each of thepixels that correspond to dynamic objects. In some implementations, thetrajectories of the dynamic objects may intersect with a trajectory ofthe vehicle.

At block 2330, a condition to maneuver the vehicle can be determined bythe computing device. For example, a motion detection algorithm can beexecuted based on the received control data to estimate an expectedmaneuver (e.g., a control signal may be expected to indicateacceleration at 50 percent for 2 seconds), determine an expected changein position or velocity based on the expected maneuver (e.g., execute avehicle load algorithm to estimate an increase in velocity expectedbased on the acceleration at 50 percent for 2 seconds), and compare theexpected change in position or velocity to the known change in positionor velocity. Based on a difference between the expected change and theknown change, the mapping can be calibrated to account for thedifference.

In some implementations, the condition to maneuver the vehicle can bebased on a potential collision with one of a static object or a dynamicobject that is moving along a predicted trajectory determined by thevisual situational awareness system. In some implementations, the visualsituational awareness system can identify one or more dynamic objectsfrom the received stream of images and determine a predicted trajectorybased on the type of the dynamic object. The condition can be triggeredbased on a relative distance between the vehicle and an estimated pointof collision. Once the estimated point of collision is less than athreshold distance, the visual situational awareness system candetermine to initiate an instruction to maneuver the vehicle.

At block 2335, control instructions are generated by the computingdevice to maneuver the vehicle responsive to determining the condition.The control instructions can correspond to a particular maneuver, whichis determined based on whether the object with which the vehicle islikely to collide is a static object or a dynamic object. In someimplementations, the control instructions can correspond to a maneuverthat is based on a predicted trajectory of the dynamic object. If thedynamic object is moving from right to left, the control instructionsmay cause the vehicle to decrease (or increase) the vehicle's altitudeor by moving from left to right by a distance sufficient to avoid thecollision. For example, an indication of a condition can be received(e.g., receive an indication of an altitude, current speed, distance todestination, distance to a leader or follower vehicle, distance to adynamic object) and the resulting maneuver performed by the vehiclerecorded based on the control data and/or change in position orvelocity. The conditions may be stored in a database and associated withmaneuvers. In some implementations, the conditions may be stored andassociated with types of dynamic objects. In some implementations, theconditions may be categorized according to types of dynamic objects,types of conditions (e.g., known object in environment, unknown objectin environment, position of vehicle relative to environment). Theconditions may be stored with a confidence level indicating a confidencethat a particular condition correlates to a particular maneuver, whichcan facilitate decision-making for executing the same maneuver torespond to the same condition.

In some implementations, control instructions are mapped to a change inposition or velocity independent of the vehicle hardware. For example,hardware-specific control instructions (e.g., a voltage to be applied toa component, a control scheme including a plurality of electricalsignals such as a pulse width modulation scheme) can be received andassociated with the change in position or velocity. As such a particularcontrol scheme (e.g., apply a voltage of 10 V for 3 seconds to a firstcomponent) can be learned or determined to correspond to a particularvehicle maneuver (e.g., accelerate at 80 percent maximum accelerationfor 3 seconds) without knowing the underlying functionality of thevehicle hardware or the vehicle motion controller. In someimplementations, the control instructions are mapped and stored in adatabase. In some implementation, the control instructions areassociated with conditions during which the maneuvers (e.g., changes inposition and/or velocity) occurred that took place at the same time asthe control instructions were received.

At block 2340, transmitting, by the computing device, the generatedcontrol instructions to the control system of the vehicle via theinterface between the computing device and the control system. Forexample, given a mapping of control instructions to a change in positionor velocity, a control signal (e.g., an electrical signal having aparticular voltage, a pulse width modulation scheme) configured to causea desired change in position or velocity can be generated. Controlinstructions associated with a maneuver that the computing devicelearned to have taken place in response to the same or a similarcondition can be retrieved. The control instructions can be retrievedfrom the various databases described herein based on conditions (e.g.,given a known or expected condition, an expected maneuver to be used torespond to the condition can be retrieved).

The control instructions can be transmitted to the control system of thevehicle via the interface. For example, an electrical signalcorresponding to the control instructions can be outputted andtransmitted via the interface. In some implementations, the controlinstructions are selectively outputted based on a decision confidence.For example, when retrieving the control instructions from a database, aconfidence can be determined by which the maneuver indicated by thecontrol instructions is expected to be an appropriate solution to thecondition (e.g., based on stored confidence values when the maneuver ismapped to the condition in the database, such as confidence valuesregarding whether the condition existed and/or whether the maneuver wasperformed responsive to the condition). The control instructions can betransmitted responsive to determining that the decision confidence isgreater than a threshold value. The threshold value can be apredetermined value. The threshold value can be adapted to the vehicleand/or the vehicle condition or a condition in an environment about thevehicle.

J. Computing Environment

FIGS. 24A and 24B depict block diagrams of a computing device 2400. Asshown in FIGS. 24A and 24B, each computing device 2400 includes acentral processing unit 2421, and a main memory unit 2422. As shown inFIG. 24A, a computing device 2400 can include a storage device 2428, aninstallation device 2416, a network interface 2418, an I/O controller2423, display devices 2424 a-2424 n, a keyboard 2426 and a pointingdevice 2427, e.g. a mouse. The storage device 2428 can include, withoutlimitation, an operating system, software, and software of visualsituational awareness system (VSAS) 800. As shown in FIG. 24B, eachcomputing device 2400 can also include additional optional elements,e.g. a memory port 2403, a bridge 2470, one or more input/output devices2430 a-2430 n (generally referred to using reference numeral 2430), anda cache memory 2440 in communication with the central processing unit2421.

The central processing unit 2421 is any logic circuitry that responds toand processes instructions fetched from the main memory unit 2422. Inmany embodiments, the central processing unit 2421 is provided by amicroprocessor unit, e.g.: those manufactured by Intel Corporation ofMountain View, Calif.; those manufactured by Motorola Corporation ofSchaumburg, Ill.; the ARM processor (from, e.g., ARM Holdings andmanufactured by ST, TI, ATMEL, etc.) and TEGRA system on a chip (SoC)manufactured by Nvidia of Santa Clara, Calif.; stand-alone ARMprocessors; the POWER7 processor, those manufactured by InternationalBusiness Machines of White Plains, N.Y.; or those manufactured byAdvanced Micro Devices of Sunnyvale, Calif.; or field programmable gatearrays (“FPGAs”) from Altera in San Jose, Calif., Intel Corporation,Xlinix in San Jose, Calif., or MicroSemi in Aliso Viejo, Calif., etc.The computing device 2400 can be based on any of these processors, orany other processor capable of operating as described herein. Thecentral processing unit 2421 can utilize instruction level parallelism,thread level parallelism, different levels of cache, and multi-coreprocessors. A multi-core processor can include two or more processingunits on a single computing component. Examples of multi-core processorsinclude the AMD PHENOM IIX2, INTEL CORE i5 and INTEL CORE i7.

Main memory unit 2422 can include one or more memory chips capable ofstoring data and allowing any storage location to be directly accessedby the microprocessor 2421. Main memory unit 2422 can be volatile andfaster than storage 2428 memory. Main memory units 2422 can be Dynamicrandom access memory (DRAM) or any variants, including static randomaccess memory (SRAM), Burst SRAM or SynchBurst SRAM (BSRAM), Fast PageMode DRAM (FPM DRAM), Enhanced DRAM (EDRAM), Extended Data Output RAM(EDO RAM), Extended Data Output DRAM (EDO DRAM), Burst Extended DataOutput DRAM (BEDO DRAM), Single Data Rate Synchronous DRAM (SDR SDRAM),Double Data Rate SDRAM (DDR SDRAM), Direct Rambus DRAM (DRDRAM), orExtreme Data Rate DRAM (XDR DRAM). In some embodiments, the main memory2422 or the storage 2428 can be non-volatile; e.g., non-volatile readaccess memory (NVRAM), flash memory non-volatile static RAM (nvSRAM),Ferroelectric RAM (FeRAM), Magnetoresistive RAM (MRAM), Phase-changememory (PRAM), conductive-bridging RAM (CBRAM),Silicon-Oxide-Nitride-Oxide-Silicon (SONOS), Resistive RAM (RRAM),Racetrack, Nano-RAM (NRAM), or Millipede memory. The main memory 2422can be based on any of the above described memory chips, or any otheravailable memory chips capable of operating as described herein. In theembodiment shown in FIG. 24A, the processor 2421 communicates with mainmemory 2422 via a system bus 2450 (described in more detail below). FIG.24B depicts an embodiment of a computing device 2400 in which theprocessor communicates directly with main memory 2422 via a memory port2403. For example, in FIG. 24B the main memory 2422 can be DRDRAM.

FIG. 24B depicts an embodiment in which the main processor 2421communicates directly with cache memory 2440 via a secondary bus,sometimes referred to as a backside bus. In other embodiments, the mainprocessor 2421 communicates with cache memory 2440 using the system bus2450. Cache memory 2440 typically has a faster response time than mainmemory 2422 and is typically provided by SRAM, BSRAM, or EDRAM. In theembodiment shown in FIG. 24B, the processor 2421 communicates withvarious I/O devices 2430 via a local system bus 2450. Various buses canbe used to connect the central processing unit 2421 to any of the I/Odevices 2430, including a PCI bus, a PCI-X bus, or a PCI-Express bus, ora NuBus. For embodiments in which the I/O device is a video display2424, the processor 2421 can use an Advanced Graphics Port (AGP) tocommunicate with the display 2424 or the I/O controller 2423 for thedisplay 2424. FIG. 24B depicts an embodiment of a computer 2400 in whichthe main processor 2421 communicates directly with I/O device 2430 b orother processors 2421′ via HYPERTRANSPORT, RAPIDIO, or INFINIBANDcommunications technology. FIG. 24B also depicts an embodiment in whichlocal busses and direct communication are mixed: the processor 2421communicates with I/O device 2430 a using a local interconnect bus whilecommunicating with I/O device 2430 b directly.

A wide variety of I/O devices 2430 a-2430 n can be present in thecomputing device 2400. Input devices can include keyboards, mice,trackpads, trackballs, touchpads, touch mice, multi-touch touchpads andtouch mice, microphones (analog or MEMS), multi-array microphones,drawing tablets, cameras, single-lens reflex camera (SLR), digital SLR(DSLR), CMOS sensors, CCDs, accelerometers, inertial measurement units,infrared optical sensors, pressure sensors, magnetometer sensors,angular rate sensors, depth sensors, proximity sensors, ambient lightsensors, gyroscopic sensors, or other sensors. Output devices caninclude video displays, graphical displays, speakers, headphones, inkjetprinters, laser printers, and 3D printers.

Devices 2430 a-2430 n can include a combination of multiple input oroutput devices, including, e.g., Microsoft KINECT, Nintendo Wiimote forthe WII, Nintendo WII U GAMEPAD, or Apple IPHONE. Some devices 2430a-2430 n allow gesture recognition inputs through combining some of theinputs and outputs. Some devices 2430 a-2430 n provides for facialrecognition which can be utilized as an input for different purposesincluding authentication and other commands. Some devices 2430 a-2430 nprovides for voice recognition and inputs, including, e.g., MicrosoftKINECT, SIRI for IPHONE by Apple, Google Now or Google Voice Search.

Additional devices 2430 a-2430 n have both input and outputcapabilities, including, e.g., haptic feedback devices, touchscreendisplays, or multi-touch displays. Touchscreen, multi-touch displays,touchpads, touch mice, or other touch sensing devices can use differenttechnologies to sense touch, including, e.g., capacitive, surfacecapacitive, projected capacitive touch (PCT), in-cell capacitive,resistive, infrared, waveguide, dispersive signal touch (DST), in-celloptical, surface acoustic wave (SAW), bending wave touch (BWT), orforce-based sensing technologies. Some multi-touch devices can allow twoor more contact points with the surface, allowing advanced functionalityincluding, e.g., pinch, spread, rotate, scroll, or other gestures. Sometouchscreen devices, including, e.g., Microsoft PIXELSENSE orMulti-Touch Collaboration Wall, can have larger surfaces, such as on atable-top or on a wall, and can also interact with other electronicdevices. Some I/O devices 2430 a-2430 n, display devices 2424 a-2424 nor group of devices can be augmented reality devices. The I/O devicescan be controlled by an I/O controller 2421 as shown in FIG. 24A. TheI/O controller 2421 can control one or more I/O devices, such as, e.g.,a keyboard 126 and a pointing device 2427, e.g., a mouse or optical pen.Furthermore, an I/O device can also provide storage and/or aninstallation medium 116 for the computing device 2400. In still otherembodiments, the computing device 2400 can provide USB connections (notshown) to receive handheld USB storage devices. In further embodiments,an I/O device 2430 can be a bridge between the system bus 2450 and anexternal communication bus, e.g. a USB bus, a SCSI bus, a FireWire bus,an Ethernet bus, a Gigabit Ethernet bus, a Fibre Channel bus, or aThunderbolt bus.

In some embodiments, display devices 2424 a-2424 n can be connected toI/O controller 2421. Display devices can include, e.g., liquid crystaldisplays (LCD), thin film transistor LCD (TFT-LCD), blue phase LCD,electronic papers (e-ink) displays, flexile displays, light emittingdiode displays (LED), digital light processing (DLP) displays, liquidcrystal on silicon (LCOS) displays, organic light-emitting diode (OLED)displays, active-matrix organic light-emitting diode (AMOLED) displays,liquid crystal laser displays, time-multiplexed optical shutter (TMOS)displays, or 3D displays. Examples of 3D displays can use, e.g.stereoscopy, polarization filters, active shutters, or autostereoscopy.Display devices 2424 a-2424 n can also be a head-mounted display (HMD).In some embodiments, display devices 2424 a-2424 n or the correspondingI/O controllers 2423 can be controlled through or have hardware supportfor OPENGL or DIRECTX API or other graphics libraries.

In some embodiments, the computing device 2400 can include or connect tomultiple display devices 2424 a-2424 n, which each can be of the same ordifferent type and/or form. As such, any of the I/O devices 2430 a-2430n and/or the I/O controller 2423 can include any type and/or form ofsuitable hardware, software, or combination of hardware and software tosupport, enable or provide for the connection and use of multipledisplay devices 2424 a-2424 n by the computing device 2400. For example,the computing device 2400 can include any type and/or form of videoadapter, video card, driver, and/or library to interface, communicate,connect or otherwise use the display devices 2424 a-2424 n. In oneembodiment, a video adapter can include multiple connectors to interfaceto multiple display devices 2424 a-2424 n. In other embodiments, thecomputing device 2400 can include multiple video adapters, with eachvideo adapter connected to one or more of the display devices 2424a-2424 n. In some embodiments, any portion of the operating system ofthe computing device 2400 can be configured for using multiple displays2424 a-2424 n. In other embodiments, one or more of the display devices2424 a-2424 n can be provided by one or more other computing devices2400 a or 2400 b connected to the computing device 2400, via the network140. In some embodiments software can be designed and constructed to useanother computer's display device as a second display device 2424 a forthe computing device 2400. For example, in one embodiment, an Apple iPadcan connect to a computing device 2400 and use the display of the device2400 as an additional display screen that can be used as an extendeddesktop. One ordinarily skilled in the art will recognize and appreciatethe various ways and embodiments that a computing device 2400 can beconfigured to have multiple display devices 2424 a-2424 n.

Referring again to FIG. 24A, the computing device 2400 can comprise astorage device 2428 (e.g. one or more hard disk drives or redundantarrays of independent disks) for storing an operating system or otherrelated software, and for storing application software programs such asany program related to the software for the VSAS 800. Examples ofstorage device 2428 include, e.g., hard disk drive (HDD); optical driveincluding CD drive, DVD drive, or BLU-RAY drive; solid-state drive(SSD); USB flash drive; or any other device suitable for storing data.Some storage devices can include multiple volatile and non-volatilememories, including, e.g., solid state hybrid drives that combine harddisks with solid state cache. Some storage device 2428 can benon-volatile, mutable, or read-only. Some storage device 2428 can beinternal and connect to the computing device 2400 via a bus 2450. Somestorage device 2428 can be external and connect to the computing device2400 via a I/O device 2430 that provides an external bus. Some storagedevice 2428 can connect to the computing device 2400 via the networkinterface 2418 over a network, including, e.g., the Remote Disk forMACBOOK AIR by APPLE. Some client devices 2400 may not require anon-volatile storage device 2428 and can be thin clients or zero clients202. Some storage device 2428 can also be used as an installation device2416, and can be suitable for installing software and programs.Additionally, the operating system and the software can be run from abootable medium, for example, a bootable CD, e.g. KNOPPIX, a bootable CDfor GNU/Linux that is available as a GNU/Linux distribution fromknoppix.net.

Computing device 2400 can also install software or applications from anapplication distribution platform. Examples of application distributionplatforms include the App Store for iOS provided by Apple, Inc., the MacApp Store provided by Apple, Inc., GOOGLE PLAY for Android OS providedby Google Inc., Chrome Webstore for CHROME OS provided by Google Inc.,and Amazon Appstore for Android OS and KINDLE FIRE provided byAmazon.com, Inc.

Furthermore, the computing device 2400 can include a network interface2418 to interface to the network 140 through a variety of connectionsincluding, but not limited to, standard telephone lines LAN or WAN links(e.g., 802.11, T1, T3, Gigabit Ethernet, Infiniband), broadbandconnections (e.g., ISDN, Frame Relay, ATM, Gigabit Ethernet,Ethernet-over-SONET, ADSL, VDSL, BPON, GPON, fiber optical includingFiOS), wireless connections, or some combination of any or all of theabove. Connections can be established using a variety of communicationprotocols (e.g., TCP/IP, Ethernet, ARCNET, SONET, SDH, Fiber DistributedData Interface (FDDI), IEEE 802.11a/b/g/n/ac CDMA, GSM, WiMax and directasynchronous connections). In one embodiment, the computing device 2400communicates with other computing devices 2400′ via any type and/or formof gateway or tunneling protocol e.g. Secure Socket Layer (SSL) orTransport Layer Security (TLS), or the Citrix Gateway Protocolmanufactured by Citrix Systems, Inc. of Ft. Lauderdale, Fla. The networkinterface 118 can comprise a built-in network adapter, network interfacecard, PCMCIA network card, EXPRESSCARD network card, card bus networkadapter, wireless network adapter, USB network adapter, modem or anyother device suitable for interfacing the computing device 2400 to anytype of network capable of communication and performing the operationsdescribed herein.

A computing device 2400 of the sort depicted in FIG. 24A can operateunder the control of an operating system, which controls scheduling oftasks and access to system resources. The computing device 2400 can berunning any operating system such as any of the versions of theMICROSOFT WINDOWS operating systems, the different releases of the Unixand Linux operating systems, any version of the MAC OS for Macintoshcomputers, any embedded operating system, any real-time operatingsystem, any open source operating system, any proprietary operatingsystem, any operating systems for mobile computing devices, or any otheroperating system capable of running on the computing device andperforming the operations described herein. Typical operating systemsinclude, but are not limited to: WINDOWS 24000, WINDOWS Server 2012,WINDOWS CE, WINDOWS Phone, WINDOWS XP, WINDOWS VISTA, and WINDOWS 24,WINDOWS RT, and WINDOWS 8 all of which are manufactured by MicrosoftCorporation of Redmond, Wash.; MAC OS and iOS, manufactured by Apple,Inc. of Cupertino, Calif.; and Linux, a freely-available operatingsystem, e.g. Linux Mint distribution (“distro”) or Ubuntu, distributedby Canonical Ltd. of London, United Kingdom; or Unix or other Unix-likederivative operating systems; and Android, designed by Google, ofMountain View, Calif., among others. Some operating systems, including,e.g., the CHROME OS by Google, can be used on zero clients or thinclients, including, e.g., CHROMEBOOKS.

The computer system 2400 can be any workstation, telephone, desktopcomputer, laptop or notebook computer, netbook, ULTRABOOK, tablet,server, handheld computer, mobile telephone, smartphone or otherportable telecommunications device, media playing device, a gamingsystem, mobile computing device, or any other type and/or form ofcomputing, telecommunications or media device that is capable ofcommunication. The computer system 2400 has sufficient processor powerand memory capacity to perform the operations described herein. In someembodiments, the computing device 2400 can have different processors,operating systems, and input devices consistent with the device. TheSamsung GALAXY smartphones, e.g., operate under the control of Androidoperating system developed by Google, Inc. GALAXY smartphones receiveinput via a touch interface.

In some embodiments, the computing device 2400 is a gaming system. Forexample, the computer system 2400 can comprise a PLAYSTATION 3, orPERSONAL PLAYSTATION PORTABLE (PSP), or a PLAYSTATION VITA devicemanufactured by the Sony Corporation of Tokyo, Japan, a NINTENDO DS,NINTENDO 3DS, NINTENDO WII, or a NINTENDO WII U device manufactured byNintendo Co., Ltd., of Kyoto, Japan, or an XBOX 360 device manufacturedby the Microsoft Corporation of Redmond, Wash., or an OCULUS RIFT orOCULUS VR device manufactured by OCULUS VR, LLC of Menlo Park, Calif.

In some embodiments, the computing device 2400 is a digital audio playersuch as the Apple IPOD, IPOD Touch, and IPOD NANO lines of devices,manufactured by Apple Computer of Cupertino, Calif. Some digital audioplayers can have other functionality, including, e.g., a gaming systemor any functionality made available by an application from a digitalapplication distribution platform. For example, the IPOD Touch canaccess the Apple App Store. In some embodiments, the computing device2400 is a portable media player or digital audio player supporting fileformats including, but not limited to, MP3, WAV, M4A/AAC, WMA ProtectedAAC, AIFF, Audible audiobook, Apple Lossless audio file formats and.mov, .m4v, and .mp4 MPEG-4 (H.264/MPEG-4 AVC) video file formats.

In some embodiments, the computing device 2400 is a tablet e.g. the IPADline of devices by Apple; GALAXY TAB family of devices by Samsung; orKINDLE FIRE, by Amazon.com, Inc. of Seattle, Wash. In other embodiments,the computing device 2400 is an eBook reader, e.g. the KINDLE family ofdevices by Amazon.com, or NOOK family of devices by Barnes & Noble, Inc.of New York City, N.Y.

In some embodiments, the communications device 2400 includes acombination of devices, e.g. a smartphone combined with a digital audioplayer or portable media player. For example, one of these embodimentsis a smartphone, e.g. the IPHONE family of smartphones manufactured byApple, Inc.; a Samsung GALAXY family of smartphones manufactured bySamsung, Inc.; or a Motorola DROID family of smartphones. In yet anotherembodiment, the communications device 2400 is a laptop or desktopcomputer equipped with a web browser and a microphone and speakersystem, e.g. a telephony headset. In these embodiments, thecommunications devices 2400 are web-enabled and can receive and initiatephone calls. In some embodiments, a laptop or desktop computer is alsoequipped with a webcam or other video capture device that enables videochat and video call.

In some embodiments, the status of one or more machines 2400 in thenetwork are monitored, generally as part of network management. In oneof these embodiments, the status of a machine can include anidentification of load information (e.g., the number of processes on themachine, CPU and memory utilization), of port information (e.g., thenumber of available communication ports and the port addresses), or ofsession status (e.g., the duration and type of processes, and whether aprocess is active or idle). In another of these embodiments, thisinformation can be identified by a plurality of metrics, and theplurality of metrics can be applied at least in part towards decisionsin load distribution, network traffic management, and network failurerecovery as well as any aspects of operations of the present solutiondescribed herein. Aspects of the operating environments and componentsdescribed above will become apparent in the context of the systems andmethods disclosed herein.

Variations to the above embodiments are also contemplated. For example,although the use case of the guidance device in UAVs is quite natural,this system and device are not limited solely to this use. Since thesystem only requires a connection to the primary vehicle controller toreceive velocity updates and to be able to issue vehicle commands, thesystem can be implemented and used for obstacle detection and collisionavoidance on, for example, rockets, missiles, manned aircraft and otherautonomous aircraft. The system can also be employed for use onautonomous naval vessels for sailing or navigation around debris orother waterborne vehicles. The system could also be used in conjunctionwith other detection systems onboard autonomous road vehicles on urbanhighways, streets or in more rural environments.

Other variations on the arrangement of the device 100 are alsocontemplated. For example, the camera 104 can be fitted either outsideor inside the case 106, or fitted directly within a side of the case106. Furthermore, the camera 104 need not be limited to optical cameras,but can include infrared or ultraviolet cameras.

In another variation of the present disclosure, the system providedherein need not result in a communication of a direct command to acontrol system. It is contemplated that the system provided herein couldbe used to communicate a suggestion, warning, or other communication toa pilot, driver, or user of a vehicle without necessarily causing thevehicle to carry out a command, such as a collision avoidance maneuver.

The scope of the claims should not be limited by the embodiments setforth in the above examples, but should be given the broadestinterpretation consistent with the description as a whole.

While this specification contains many specific implementation details,these should not be construed as limitations on the scope of anydisclosures or of what can be claimed, but rather as descriptions offeatures specific to particular embodiments of particular aspects.Certain features described in this specification in the context ofseparate embodiments can also be implemented in combination in a singleembodiment. Conversely, various features described in the context of asingle embodiment can also be implemented in multiple embodimentsseparately or in any suitable subcombination. Moreover, althoughfeatures can be described above as acting in certain combinations andeven initially claimed as such, one or more features from a claimedcombination can in some cases be excised from the combination, and theclaimed combination can be directed to a subcombination or variation ofa subcombination.

Similarly, while operations are depicted in the drawings in a particularorder, this should not be understood as requiring that such operationsbe performed in the particular order shown or in sequential order, orthat all illustrated operations be performed, to achieve desirableresults. In certain circumstances, multitasking and parallel processingcan be advantageous. Moreover, the separation of various systemcomponents in the embodiments described above should not be understoodas requiring such separation in all embodiments, and it should beunderstood that the described program components and systems cangenerally be integrated in a single software product or packaged intomultiple software products.

References to “or” may be construed as inclusive so that any termsdescribed using “or” may indicate any of a single, more than one, andall of the described terms. References to at least one of a conjunctivelist of terms may be construed as an inclusive OR to indicate any of asingle, more than one, and all of the described terms. For example, areference to “at least one of ‘A’ and ‘B’” can include only ‘A’, only‘B’, as well as both ‘A’ and ‘B’.

Thus, particular embodiments of the subject matter have been described.In some cases, the actions recited in the claims can be performed in adifferent order and still achieve desirable results. In addition, theprocesses depicted in the accompanying figures do not necessarilyrequire the particular order shown, or sequential order, to achievedesirable results.

What is claimed is:
 1. A method for generating visual situationalawareness models, comprising: receiving, by a computing device includingone or more processors, from an image capture device mounted on amovable object, a sequence of images captured over a period of time,each image of the sequence of images including a plurality of pixels;determining, by the computing device, motion data comprising velocityvalues or acceleration values of the movable object that correspond towhen each of the sequence of images were captured by the image capturedevice; tracking, by the computing device, movement of at least a subsetof the plurality of pixels of respective images across the sequence ofimages; estimating, by the computing device responsive to tracking thesubset of the plurality of pixels across the sequence of images andusing the velocity values or the acceleration values of the movableobject corresponding to each of the sequence of images, depth values ofthe tracked subset; generating, by the computing device, a 3D world mapbased on the estimated depth values of the tracked subset; identifying,by the computing device, from the sequence of images, a plurality ofpoints of interest mapped to respective pixels of the plurality ofpixels, the plurality of points of interest identified to track dynamicobjects; tracking, by the computing device, across the sequence ofimages, for each identified point of interest of the plurality of pointsof interest, movement of the point of interest; identifying, from thetracked movements of the plurality of points of interest, one or moredynamic objects based on comparing actual positions of the respectivepoint of interest in the sequence of images relative to expectedpositions of the respective point of interest calculated based on thevelocity values or the acceleration values of the movable objectcorresponding to each of the sequence of images; and generating, by thecomputing device, a visual situational awareness model using thegenerated 3D world map and the one or more dynamic objects.
 2. Themethod of claim 1, wherein determining the motion data of the movableobject includes receiving the motion data from the movable object via acommunication link.
 3. The method of claim 1, wherein determining themotion data of the movable object includes determining motion data via avelocity determination module executing on the computing device, thevelocity determination module using triangulation.
 4. The method ofclaim 1, wherein tracking, by the computing device, movement of at leastthe subset of the plurality of pixels of the respective images acrossthe sequence of images includes tracking, by the computing device,movement of at least the subset of the plurality of pixels of therespective images across the sequence of images using a respectiveparameter value of a respective pixel of the subset of the plurality ofpixels.
 5. The method of claim 1, wherein the depth values include anabsolute depth value.
 6. The method of claim 1, wherein the depth valuesare estimated without loop closure.
 7. The method of claim 1, whereinthe 3D world map includes a plurality of points, each pointcorresponding to a respective pixel and a respective estimated depthvalue of the respective pixel.
 8. The method of claim 1, whereinidentifying points of interest includes applying a contrast filter toidentify pixels having a contrast ratio relative to pixels within apredetermined distance of the respective pixel.
 9. The method of claim1, wherein estimating the depth values of the tracked subset of pixelsincludes determining, for each pixel of the tracked subset of pixels, adistance the pixel moved across the sequence of images and estimating adepth value of the pixel based on the determined distance the pixelmoved.
 10. The method of claim 1, further comprising generatinginstructions to perform an action responsive to the one or more dynamicobjects.
 11. A system for generating visual situational awarenessmodels, comprising: at least one processor; a memory coupled to the atleast one processor; an image receiver configured to receive, from animage capture device mounted on a movable object, a sequence of imagescaptured over a period of time, each image of the sequence of imagesincluding a plurality of pixels; a motion data identifier configured todetermine motion data comprising velocity values or acceleration valuesof the movable object that correspond to when each of the sequence ofimages were captured by the image capture device; a pixel movementtracker configured to track movement of at least a subset of theplurality of pixels of respective images across the sequence of images;a depth value estimator configured to estimate, responsive to trackingthe subset of the plurality of pixels across the sequence of images andusing the velocity values or the acceleration values of the movableobject corresponding to each of the sequence of images, depth values ofthe tracked subset; a 3D world map generator configured to generate a 3Dworld map based on the estimated depth values of the tracked subset; apoint of interest identifier configured to identify, from the sequenceof images, a plurality of points of interest mapped to respective pixelsof the plurality of pixels, the plurality of points of interestidentified to track dynamic objects; a point of interest movementtracker configured to track, across the sequence of images, for eachidentified point of interest of the plurality of points of interest,movement of the point of interest; a dynamic object identifierconfigured to identify, from the tracked movements of the plurality ofpoints of interest, one or more dynamic objects based on comparingactual positions of the respective point of interest in the sequence ofimages relative to expected positions of the respective point ofinterest calculated based on the velocity values or the accelerationvalues of the movable object corresponding to each of the sequence ofimages; and a visual situational awareness model generator configured togenerate a visual situational awareness model using the generated 3Dworld map and the one or more dynamic objects.
 12. The system of claim11, wherein the motion data is received from the movable object via acommunication link.
 13. The system of claim 11, wherein the motion dataidentifier includes a velocity determination module configured todetermine the velocity values of the movable object using triangulation.14. The system of claim 11, wherein the pixel movement tracker isconfigured to track movement of at least the subset of the plurality ofpixels of the respective images across the sequence of images using arespective parameter value of a respective pixel of the subset of theplurality of pixels.
 15. The system of claim 11, further comprising acollision avoidance manager configured to generate instructions toperform an action responsive to the one or more dynamic objects.
 16. Thesystem of claim 11, wherein the depth values are estimated without loopclosure.
 17. The system of claim 11, wherein the plurality of points ofinterest are identified by applying a contrast ratio filter to identifypixels having a contrast ratio relative to pixels within a predetermineddistance of the respective pixel.
 18. The system of claim 11, whereinthe depth value estimator is configured to determine, for each pixel ofthe tracked subset of pixels, a distance the pixel moved across thesequence of images and estimate a depth value of the pixel based on thedetermined distance the pixel moved.